11

Network QuarantineNetwork QuarantineAtAt

Cornell UniversityCornell University

Steve SchusterSteve Schuster

Director, Information Security Director, Information Security OfficeOffice

2

OverviewOverview

►Cornell’s incident response strategyCornell’s incident response strategy► Introduction to Network QuarantineIntroduction to Network Quarantine►Review of Scan at Registrations Review of Scan at Registrations

System (SARS)System (SARS)►Post Mortem (What we did Post Mortem (What we did

intelligently)intelligently)►Future considerationsFuture considerations

3

Organizational StructureOrganizational Structure► Contact CenterContact Center

Part of Customer Services and MarketingPart of Customer Services and Marketing Address end user supportAddress end user support

► Patch supportPatch support► Virus remediationVirus remediation

► Network Operations Center (NOC)Network Operations Center (NOC) Part of Systems and OperationsPart of Systems and Operations Initial security triageInitial security triage Incident responseIncident response

► BlocksBlocks► NotificationsNotifications

► IT Security OfficeIT Security Office Development of operational proceduresDevelopment of operational procedures Technical solutionsTechnical solutions Backline supportBackline support

4

Some Security Challenges at Some Security Challenges at CornellCornell

►A general openness and decentralization A general openness and decentralization leads to a larger number of incidentsleads to a larger number of incidents

►Responding to incidents can be staff Responding to incidents can be staff intensiveintensive

►Unmanaged (students) systems arrive Unmanaged (students) systems arrive on our network several times each yearon our network several times each year

► Incident notification is a challengeIncident notification is a challenge►Wide range of end user support needsWide range of end user support needs

5

Responding to IncidentsResponding to Incidents

► Security Office will react and contain campus Security Office will react and contain campus systems that are compromised or highly vulnerablesystems that are compromised or highly vulnerable

► NOC had a mix of tools and manual processes for NOC had a mix of tools and manual processes for opening case, notifying impacted parties and opening case, notifying impacted parties and implementing containmentimplementing containment

► Security Office often sends NOC containment Security Office often sends NOC containment requests that were tedious to service with current requests that were tedious to service with current toolstools

► Response to wide range security issues put much Response to wide range security issues put much strain on Contact Centerstrain on Contact Center

► Current mechanism for containment was not fully Current mechanism for containment was not fully effective and didn’t work in some environmentseffective and didn’t work in some environments

6

Network QuarantineNetwork Quarantine

► ObjectivesObjectives Provide better end user communication based Provide better end user communication based

upon observed incidentupon observed incident Articulate self-remediation information and Articulate self-remediation information and

requirements when appropriaterequirements when appropriate Improve cost effectiveness of security supportImprove cost effectiveness of security support

► NocNoc► Contact CenterContact Center

More effective system isolationMore effective system isolation Better incident tracking and remediation for local Better incident tracking and remediation for local

support providerssupport providers Quicker/escalated response for critical systemsQuicker/escalated response for critical systems

7

Network QuarantineNetwork Quarantine(Basic Features)(Basic Features)

► The right action is taken depending upon type of systemThe right action is taken depending upon type of system ““Registration” 10 spaceRegistration” 10 space DMZ blockedDMZ blocked ““Critical system” notificationCritical system” notification

► Response for systems identified as critical is escalated to Security Response for systems identified as critical is escalated to Security Office and appropriate local support providerOffice and appropriate local support provider

► Incidents can be created, modified and closed via web and socket Incidents can be created, modified and closed via web and socket interfacesinterfaces Latter allows batch and automationLatter allows batch and automation

► NQ interacts with Vantive, creating new case when incident NQ interacts with Vantive, creating new case when incident openedopened

► Modifications to an incident trigger e-mail to user, net admin and Modifications to an incident trigger e-mail to user, net admin and updates to Vantiveupdates to Vantive

► Specific incident remediation information provided for end users Specific incident remediation information provided for end users ► With appropriate credentials, CIT personnel, including Contact With appropriate credentials, CIT personnel, including Contact

Center, and campus system administrators can search for and Center, and campus system administrators can search for and review incidentsreview incidents

8

Network QuarantineNetwork Quarantine(Incident Types)(Incident Types)

9

Network QuarantineNetwork Quarantine(Incident Types)(Incident Types)

10

Network QuarantineNetwork Quarantine(Incident Messages)(Incident Messages)

11

Network QuarantineNetwork Quarantine(Incident Containment)(Incident Containment)

12

Network QuarantineNetwork Quarantine(Incident Remediation)(Incident Remediation)

13

Network QuarantineNetwork Quarantine(User’s View)(User’s View)

14

Network QuarantineNetwork Quarantine(User’s View)(User’s View)

15

Network QuarantineNetwork Quarantine(User’s View)(User’s View)

128.XXX.XXX.XXX

16

Network QuarantineNetwork Quarantine(Specific Features)(Specific Features)

► For each new incidentFor each new incident New incident type for trackingNew incident type for tracking Establishment of resolution requirementsEstablishment of resolution requirements Incident specific message to usersIncident specific message to users

► Users receive much better communicationUsers receive much better communication► Self-release feature Self-release feature

Users are able correct the issue Users are able correct the issue Save staff time at the Contact CenterSave staff time at the Contact Center

► Process automation, better user Process automation, better user communication and self-release has saved communication and self-release has saved moneymoney

17

Incident Response CostsIncident Response Costs

►Virus remediation costs/incidentVirus remediation costs/incident Contact Center – Average 10 minutesContact Center – Average 10 minutes NOC – Average 3 minutesNOC – Average 3 minutes

►System compromise costs/incidentSystem compromise costs/incident Contact CenterContact Center

►Simple support -- 20 minutesSimple support -- 20 minutes►Full rebuild – 1-4 hoursFull rebuild – 1-4 hours

NOC – Average NOC – Average ►Average 5 minutesAverage 5 minutes

18

Network Quarantine Network Quarantine (Cost Savings)(Cost Savings)

►Virus remediation costs/incidentVirus remediation costs/incident Contact Center – Same but many self-Contact Center – Same but many self-

releaserelease NOC –under 1 minuteNOC –under 1 minute

►System compromise costs/incidentSystem compromise costs/incident Contact CenterContact Center

►Simple support -- 20 minutesSimple support -- 20 minutes►Full rebuild – 1-4 hoursFull rebuild – 1-4 hours

NOC – Average NOC – Average ►Under 1 minuteUnder 1 minute

19

Scan at Registration SystemScan at Registration System(SARS)(SARS)

►All on-campus student computers were All on-campus student computers were automatically scanned upon automatically scanned upon registrationregistration

►ObjectsObjects Drastically reduce the number of infected Drastically reduce the number of infected

or compromised student systems coming or compromised student systems coming to campusto campus

Promote better security practicesPromote better security practices

20

Enabling Features of NQ that Enabling Features of NQ that Supported SARSSupported SARS

►Automation of containment and Automation of containment and remediationremediation

►Redirection to Network Quarantine Redirection to Network Quarantine infrastructureinfrastructure

►Articulated steps to support self-Articulated steps to support self-remediationremediation

► Incident tracking Incident tracking

21

Scan at Registration System Scan at Registration System (SARS)(SARS)

►Requirements for ResNet registrationRequirements for ResNet registration Each computer system must be registered Each computer system must be registered

with a valid NetIDwith a valid NetID Each computer must be configured to a Each computer must be configured to a

minimum set of security standardsminimum set of security standards►No open writable filesharesNo open writable fileshares►All administrative accounts must have a All administrative accounts must have a

passwordpassword►Must be patched Must be patched

22

Student Registration ProcessStudent Registration Process

► Every on-campus student went through the follow Every on-campus student went through the follow processprocess Plug into network and get redirected to ResNet Plug into network and get redirected to ResNet

Registration pageRegistration page Authentication with NetID and fill in necessary information Authentication with NetID and fill in necessary information

for registrationfor registration Wait 90 seconds for registration to complete and system Wait 90 seconds for registration to complete and system

check to occurcheck to occur If the system passed all three testsIf the system passed all three tests

► Registration competeRegistration compete ElseElse

► Redirected to NQRedirected to NQ► Informed of the problem and provided directions for Informed of the problem and provided directions for

remediationremediation► Rescan upon completion of remediationRescan upon completion of remediation► RepeatRepeat

23

Scan at Registration Scan at Registration StatisticsStatistics

► Approximately 6500 systems scanned over Approximately 6500 systems scanned over move in weekendmove in weekend

► Of all systems scannedOf all systems scanned 65% were probably firewalled65% were probably firewalled 35% were not firewalled35% were not firewalled

► 25% were clean25% were clean► 10% had at least one of the three problems10% had at least one of the three problems

► Close to 12% of the systems had at least Close to 12% of the systems had at least one problem (780)one problem (780)

► Around 85% of all quarantined students Around 85% of all quarantined students were able to perform self remediationwere able to perform self remediation

24

Network QuarantineNetwork QuarantineOn-Boarding MetricsOn-Boarding Metrics

0

100

200

300

400

500

600

700

800

900

Date

Number of VulnerableSystems

Number of Open Cases

25

Post MortemPost Mortem

► Gaining early support from Contact Center Gaining early support from Contact Center and NOC was an absolute requirementand NOC was an absolute requirement

► Can’t under estimate the stress of move in Can’t under estimate the stress of move in weekend (the parent affect)weekend (the parent affect)

► Trust is important but “bail out” features go Trust is important but “bail out” features go further further If the scanning or quarantine infrastructure failed If the scanning or quarantine infrastructure failed

registration would continue as beforeregistration would continue as before If the Contact Center could not support the If the Contact Center could not support the

demands of quarantined students all could be demands of quarantined students all could be released immediatelyreleased immediately