Talk at the Security Workshop, GridKA Summerschool 2010

Incidentreporting

S. Freitag, F.Feldhaus

Before youreport

IncidentScenarios

Incidenthandling

Incident reportingGridKa Summer School 2010

Stefan Freitag, Florian Feldhaus

Robotics Research InstituteTU Dortmund

September 10, 2010

Incidentreporting


Before youreport

IncidentScenarios

Incidenthandling

Contents

1 Before you report

2 Incident Scenarios

3 Incident handling

Incidentreporting


Before youreport

IncidentScenarios

Incidenthandling

Do you know....?

Security Incident Response Policy1

objective: ensure that all incidents are investigated as fullyas possible and that sites promptly report intrusions.

As a grid participant, you agree to

report suspected security incidents that have impact orrelationship to grid resources, services, or identitiesrespond to and investigate incident reports regardingresources, services, or identities for which you areresponsibleperform appropriate investigations and forensics and sharethe results with the incident coordinatorfollow the incident response procedure

Next question: what is the incident response procedure?1https://edms.cern.ch/document/428035/7

https://edms.cern.ch/document/428035/7

Incidentreporting


Before youreport

IncidentScenarios

Incidenthandling

EGEE incident response procedure2

Audience

grid site security officers and site administrators

Definition of security incident

The act of violating an explicit or implied security policy

Definition of actions for the case of a security incident

More on this in a few minutes . . .

2https://edms.cern.ch/document/867454

https://edms.cern.ch/document/867454

Incidentreporting


Before youreport

IncidentScenarios

Incidenthandling

Security incident - scenario A (2009)

Some grid sites allow gsissh-based access to VoBoxes (e.g.for VO software managers)

On a VoBox Grid users are mapped to local accounts

Initial step for an attacker

gain access to user credentials (certificate or proxy)

What happens next ?

Connect to VoBox using stolen credentials

Running e.g. a kernel exploit to gain root privileges

Incidentreporting


Before youreport

IncidentScenarios

Incidenthandling

Security incident - scenario A (2009)

# sh −x wunderbar empor ium . sh[ . . . ][+] got r i n g 0 ![+] d e t e c t ed 2 .6 s t y l e 4k s t a c k s[+] D i s ab l e d s e c u r i t y o f : noth ing , what an

i n s e c u r e machine ![+] Got r oo t !

sh−3.00# idu id=0( r oo t ) g i d=0( r oo t ) g roups =64004( hepcg )

con t e x t=u s e r u : s y s t em r : i n i t r c t

Incidentreporting


Before youreport

IncidentScenarios

Incidenthandling

Security incident - scenario B (2010)

Department A The Grid

Incidentreporting


Before youreport

IncidentScenarios

Incidenthandling



CERTIFICATEX.509

CERTIFICATEX.509

Incidentreporting


Before youreport

IncidentScenarios

Incidenthandling



CERTIFICATEX.509

CERTIFICATEX.509

Alienattacker

Incidentreporting


Before youreport

IncidentScenarios

Incidenthandling


The Grid

CERTIFICATEX.509

Alienattacker

stolen

Incidentreporting


Before youreport

IncidentScenarios

Incidenthandling

Incident handling

For the next slides please keep in mind:

The red block describes actions required by the EGEE IncidentResponse Procedure document

The blue block contains information about actions carried outduring a security incident at the Grid resource in Dortmund

Down here you will find additional information, e.g. max.response times

Incidentreporting


Before youreport

IncidentScenarios

Incidenthandling

Incident handling

First action

Inform immediately your local security team and your ROCSecurity Contact

Action

Sent E-Mail to Ursula Epting

Read Incident response procedure

Informed 2nd site security officer and local security team

max. 4 hours or

Incidentreporting


Before youreport

IncidentScenarios

Incidenthandling

Incident handling

Response procedure

In case no support is shortly available [...] try to contain theincident. For instance by unplugging the network cableconnected to the host. Do NOT reboot or power off the host.

Action

Disconnected affected workernodes from network

Incidentreporting


Before youreport

IncidentScenarios

Incidenthandling

Incident handling

Response procedure

Assist your local security team and your ROC Security Contactto confirm and investigate the incident. Announce the incidentto all the sites.

Actions

Send a heads-up e-mail (template: next slide)

Arranged meeting with local security team

Network guys were asked to check logs

max. 4 hours (Announcement)

Incidentreporting


Before youreport

IncidentScenarios

Incidenthandling

Heads-up E-mail

** PLEASE DO NOT REDISTRIBUTE ** EGEE-<DATE> (ex: EGEE-20090531)

** This message is sent to the EGEE CSIRTs and must NOT be publicly archived **

Dear CSIRTs,

It seems a security incident has been detected at <your site>.

Summary of the information available so far:

Ex: A malicious SSH connection was detected from XXXXX. The extent of the

incident is unclear for now, and more information will be published in the coming

hours as forensics are progressing at our site. However, all sites should check for

successful SSH connection from XXXXX as a precautionary measure.

Incidentreporting


Before youreport

IncidentScenarios

Incidenthandling

Incident handling

Response procedure

Report a downtime for the affected hosts on the GOCDB

→ Send an EGEE broadcast announcing the downtime forthe affected hosts Use ”Security operations in progress” asthe reason with no additional detail both for the broadcastand the GOCDB.

Actions

Created downtime for possibly affected hosts udo-ce01/udo-dcache01

max. 1 day after discovery

Incidentreporting


Before youreport

IncidentScenarios

Incidenthandling

Incident handling

Response procedure

Perform appropriate forensics and take necessary actions toprevent further damage

Identify and kill suspicious process(es) as appropriate, butaim at preserving the information they could havegenerated

If it is suspected that some grid credentials have beenabused or compromised, you MUST ensure the relevantaccounts become suspended

If it is suspected that some grid credentials have beenabused, you MUST ensure that the relevant VOmanager(s) have been informed.

Incidentreporting


Before youreport

IncidentScenarios

Incidenthandling

Incident handling

Response procedure

Perform appropriate forensics and take necessary actions toprevent further damage

If it is suspected that some grid credentials have beencompromised, you MUST ensure that the relevantcertification authority gets informed.

If needed, seek for help from your local security team orfrom your ROC Security Contact

Incidentreporting


Before youreport

IncidentScenarios

Incidenthandling

Incident handling

Action

Banned affected users on our compute elements by addingtheir DN to the blacklist in/opt/glite/etc/lcas/ban users.db

E-Mail to VO manager regarding compromised user

Contacted the certification authority

Incidentreporting


Before youreport

IncidentScenarios

Incidenthandling

Incident handling

Response procedure

As part of the security incident resolution process, sites areexpected to report the following information:

affected hosts and hosts used as entry point to the site

remote IP address(es) of the attacker

evidence of the compromise, including timestamps

what was lost, details of the attack

list of other sites possibly affected (if available)

possible vulnerabilities exploited by the attacker (ifavailable)

actions taken to resolve the incident

Incidentreporting


Before youreport

IncidentScenarios

Incidenthandling

Incident handling

Response procedure

Tracked down the UI that was used by the attacker for jobsubmission (checking logs of batchsystem, ComputeElement, . . . )

Analyzed netflow to/fro affected workernode

Analyzed executables deployed by the attacker

Updated incident report regularly

Incidentreporting


Before youreport

IncidentScenarios

Incidenthandling

Incident handling

Response procedure

Coordinate with your local security team and your ROCSecurity Contact to send an incident closure report includinglessons learnt and measures taken to prevent future incidents.

Actions

Preparation and submission of final report

max. 1 months

Incidentreporting


Before youreport

IncidentScenarios

Incidenthandling

Incident handling

Response procedure

Restore the service, and if needed, send an EGEE broadcast,update the GOCDB, service documentation and procedures toprevent recurrence as necessary

Actions

Re-installation of affected workernode

Safety tuning

Incidentreporting


Before youreport

IncidentScenarios

Incidenthandling

Thanks for your attention!

Education

Talk at the Security Workshop, GridKA Summerschool 2010