Upload
drupal-project
View
2.046
Download
2
Embed Size (px)
Citation preview
Life after the hack
OSInetFrédéric G. MARAND (fgm)
2/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
Topics• 1 Intro : setting the stage• 2 Snapshotting• 3 Maintaining presence• 4 Crisis communication• 5 Rebuild, don’t repair• 6 Using forensics tools• 7 Back online
4/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
1.1 Some fact checking first• In this room …• Who has been hacked already ?• Who feels ready to face a hacked server ?• Who actually has a contingency plan ?• Who read node 2365547 ?
5/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
6/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
1.2 Can you say that again ?
I.A.N.A.L.So be sure to get one !
7/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
1.3 Whence do I speak ?• Drupal org member since 2005 (fgm)• Drupal consultant, not a site building agency• Worked on fixing broken (in) sites since 2008• Auditing• Fixing technical flaws• Addressing intrusions / exploits
• Mostly Media and Government sites (.fr)• Provisional member of the Security Team
8/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
1.4 Setting the stage • 10:00 The daily scrum has just begun.• 10:01 Phones rings : someone noticed your site
has been defaced and is warning you• 10:02 Twitter and Reddit start buzzing• 10:05 Phones ring all over the place, with
journalists and the various C-level execs onthe other end, your mailbox is filling withwarnings• What is your next step ?
9/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
1.5 Get ready• Pad 1 : discovery log• all your work steps• all your findings / observations• with timestamps and numbers
• Pad 2 : remedies ideas• cross-refer pad 1 numbers• all your ideas for fixing the breach• all your ideas for further hardening
11/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
2.1 Forensic copy : why ?• First temptation : restore and resume• But you’re still vulnerable• So you need to diagnose
• Analyzing means modifying• So preserve the « crime scene »• Snapshot everything
12/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
2.2 Snapshots : pull the plug• Prevents interference• Shutdown
handlers, SIGPWR• Self-destructing
code on networkloss
• Easy on VMs
But…• Bare remote servers• Further data loss• Journaled FS• Databases
• Service interruption
13/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
2.3 Snapshots : what ?Not just the main DB
• Reverse Proxy logs• Web fronts• DB servers• File servers
And also…
• External logs (SaaS)• External transactions• IDS/firewall logs
The site may just be an attack vector
15/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
3.1 Maintaining presence 1
• Yes• Don’t tip off
hackers• Keep generating
short-term value
• No• Increasing
damage• Responsibility• Legal• Financial• Moral
As though intrusion had not been detected
16/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
3.2 Attacker workflowEvolved• Break in• Dig for gold• Implant zombie• Wait for implant
migration to archives• Activate• Profit
• Alt : Need for Speed• Use exploit ASAP• While it lasts• Usually least loss
• Alt : hidden steal• Valuable content• Identity data• Close the door
17/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
3.3 Maintaining presence 2
• Limited static site• Best with prior
work• Minimal subset• Possibly taken
from RP cache• Very little load : can
run on RP heads
• Working limited site• Alternate infra• Alternate tech
• Updates ?• Content created
during this step
Safe fallback mode
18/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
3.4 Maintaining presence 3When all else fails
• Social networks• Always there• Also authoritative for audience
• Still needs some preparation :• Accounts access• Include them in long-term communication
20/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
4.1 Communicating : from tech• Stakeholders• Chain up to CxO level in most cases• Prepare next steps, do not overreach
• Fear of reprisal ? Gag orders, SLAPP…• Protection• France : whistleblower protection (Sapin 2)• Italy : Dec. 385 01/09/93 sect 52bis (banks)• US : Anti-SLAPP• Many other countries have similar rules
21/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
4.2 : Communication : C-level• Legal counsel (first)• Crisis Management specialists• Law enforcement• EU countries typically have specialized
units for « cybercrime »• Other sites• On same server• On same network• Online business partners
22/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
4.3 Communication : privacy• In many cases personal data leaks• will happen, or...• unprovable they did not happen
• Operational constraints• Commerce : PCI/DSS (12 steps etc)• Health : (US) HIPAA Subtitle D E2.80.93
• Public image damage control• A french example
24/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
5.1 Rebuild : keep, rollback or ?• Restore and restart same ?• Still just as vulnerable
• Keep and fix ?• lots of time and effort reviewing• never completely trusted : not just Drupal
• Throw away ?• Event sites, past lines of biz, post-M&A...• Can a static version suffice ?• From RP snapshots : recent content
25/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
5.2 Rebuild : restore• Needs backups from before the hack• Do you know when it happened ?• Remember attacker workflow « wait »
• GFS, continuous incremental, 15 min ?• How much can you lose ?
• FLOSS solutions : Amanda, Bacula, custom• Unprepared emergency ?• Preproduction, CI builds...
26/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
5.3 Rebuild : sources + export• Easy and reliable, but assumes :• Code-driven development process• Reliable data export system in place• Flat content exports• Content + assets repositories
• Still need to add the fixes• Delay can be a problem on high-volume sites• Bulk handling, Incremental loading
27/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
5.4 Rebuild : other cases• Ad hoc « traditional » build process• Longer, less reliable• Too long to be a chance to fix the process
• From scratch• Too long in most cases• Do it as a complement after the fix• Not NOW
29/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
6 Forensics : switching hats
30/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
6.1 Forensics : first, think !• How did you become aware of hack ?• What did it take to succeed ?• Cast your net wide, think big• « Unlikely » vs « impossible »
• Priority :• Easiest attacks first• OWASP 10• GIYF : search your Pad 1 patterns
31/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
6.2 Forensics : keep in mind• /anything/ may be erased after success• But most of the time, not /everything/ will
• Anything you do leaves its own traces• Work on copies of the snapshots• You can restart from fresh copies anytime
• There maybe more than one exploit
32/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
6.3 Forensics : classics• Code files :• lax permissions• filesystem traversal issues• Remote payload execution by upload
• Nginx without extra hardening• .htaccess won’t do much good
• In-DB PHP• PHP module• Eval-uated code
33/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
6.4 Forensics : non-Drupal• Filesystem : • <user>/www-data outside /sites• www-data/www-data suspicious• x bit on files below docroot
• timestamps • outside sites/*/files = install• exploits > install
• meld with fresh build from sources• Also check outside docroot
34/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
6.5 Forensics : Drupal modules• Code signing/diffing :• Hacked!• D7 : md5check, file_integrity
• Finding DB PHP• QA (github)
• Misc• security_review
35/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
6.6 Forensics : DB• Quick wins :• users.email!= users.init• review roles, accounts with admin roles• On corp. sites, users.email domains• match users accounts with SSO data
• Diff DB snapshot with live• Especially menu_router :
file_put_contents, assert• Altova DatabaseSpy content compare
36/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
6.7 Forensics : sessions• Sessions should be in persistent storage• Remember when you pulled the plug• Were your sessions in Memcache ?
• sessions.timestamp vs users_field_data :created/changed/access/login• for intranets : sessions.hostname
37/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
6.8 Forensics : logs• You use off-site logs, right ?• SaaS : Loggly, Logmatic, Logsene, Logz.io,
Papertrail, Scalyr….• Remote ELK
• On site ?• dblog {watchdog}• syslog → follow the redirects• mongodb_watchdog
• Application/WS logs
38/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
6.9 Forensics : sleuth tools• Software• Guidance
Software : Encase• AccessData :
UltimateForensics Toolkit(FTK)
• Consider certifiedconsultants
40/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
7.1 Live again : restoring prod• Recheck Pad 1 findings vs new build• Usually, reset passwords. On D7 :• update users set pass = concat('ZZZ', sha(concat(pass, md5(rand()))));
• Prepare marketing/social copy
41/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
7.2 L8R : future-readiness
42/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
7.3 L8R : disaster prevention• Developer education on security• Security Team mailing list• https://twitter.com/drupalsecurity • https://www.drupal.org/security/rss.xml• http://crackingdrupal.com/
43/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
7.4 L8R : disaster prevention• Security process• Analyse sec. releases to understand fixes• Look for similar flaw in custom code• Take part in contrib for more expertise
• Quality process• Systematic peer code reviews• Code-driver maintenance + dev process• Automatic quality tools in CI• Contrib updates scheduling
44/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
7.5 Continuous improvement• You can’t improve what you don’t measure• Get time metrics from Pad 1
• Build contigency plan from Pad 2• Plan for periodic intrusion simulations
45/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr