CTSSR Major Post-Event Debriefings-Willful Disruptions: Cybersecurity

Michael P. Lewis, Executive DirectorJuly 11, 2018

February 2018

Insert pictureWhat happened to your files?

All your files encrypted with RSA-20148 encryption, For more information search in Google ‘RSA Encryption’

How to recover files?

RSA is a asymmetric cryptographic algorithm, You need one key for encryption and one key for decryptionSo you need Private key to recover your filesIt’s not possible to recover your files without private key

2018 CDOT Cyber Event

• February 18, 2018, a bad actor entered the system.

• By February 23rd, the SamSamRansomware Attack on CDOT had severely crippled operations, with thousands of employees unable to access email, electronic files, or computer applications.

• Actions taken to respond and recover from the attack serve as valuable insight for organizations learning to prepare themselves against this serious, modern-era threat.

Insert picture

“IMPORTANT and URGENT: Please Log off and shut down your computers ASAP due to a virus running through the state system.”

– Kerry Cataldo, CDOT

Progression of the Attack

Phase 1: Threat actor conducts brute force attack to gain administrator privileges and downloads tool to enumerate domain controller.

DownloadedTool

Online ServerXX.XXX.XX.X

IP Address: XX.XXX.XX.XX

Phase 1: Threat actor conducts brute force attack to gain administrator privileges and downloads tool to enumerate domain controller.

Phase 2: From the Online Server the threat actor deploys a legitimate domain admin tool.

DownloadedTool

Online ServerXX.XXX.XX.X

Domain Controller 01

IP Address: XX.XXX.XX.XX

Progression of the Attack

Phase 1: Threat actor conducts brute force attack to gain administrator privileges and downloads tool to enumerate domain controller.

Phase 2: From the Online Server the threat actor deploys a legitimate domain admin tool.

Phase 3: Threat actor gathers host names from Domain Controller 01.

DownloadedTool

Online ServerXX.XXX.XX.X

Domain Controller 01

6,495 Host names

IP Address: XX.XXX.XX.XX

Progression of the Attack

Phase 1: Threat actor conducts brute force attack to gain administrator privileges and downloads tool to enumerate domain controller.

Phase 2: From the Online Server the threat actor deploys a legitimate domain admin tool.

Phase 3: Threat actor gathers host names from Domain Controller 01.

Phase 4: Threat actor validates access on endpoint device(s).

DownloadedTool

Online ServerXX.XXX.XX.X

Domain Controller 01

6,495 Host names

Endpoint Laptop

IP Address: XX.XXX.XX.XX

Progression of the Attack

Phase 1: Threat actor conducts brute force attack to gain administrator privileges and downloads tool to enumerate domain controller.

Phase 2: From the Online Server the threat actor deploys a legitimate domain admin tool.

Phase 3: Threat actor gathers host names from Domain Controller 01.

Phase 4: Threat actor validates access on endpoint device(s).

Phase 5: Threat actor installs service to execute SamSam malware and infecting numerous network systems

• Install• Execute

Malware

CDOT Environment

Domain Controller 01

Progression of the Attack

• One of the first DOT cyber attacks nationwide.

• Lack of guidance to help CDOT on how to respond and recover quickly

• CDOT does have previous experience with responding quickly to natural disasters – floods, fires, etc.

• Utilized the same Incident Command Structure (ICS) and Incident Management Team (IMT)

Incident Command Setup

• Original Command Structure quickly expanded to a Unified Command to access vulnerabilities to statewide network

• Managed by the State Office of Emergency Management

• Command included the ColoradoNational Guard Cyber Team

Statewide Unified Command

Unified Command

CDOT

ICC

OIT / CoNG

Containment Eradication System Recovery

Unified Command

Support

Balancing competing priorities between the technical (OIT) and internal (CDOT) response teams:

• OIT Priority – Contain and eradicate malware, recover the system

• CDOT Priority – Maintain and continue functioning business aspect of the Department

• Cannot be sustained without computer infrastructure operational and available to staff

Business v. Technical Language

Balancing Business and Technical

• Maintaining “Business as Usual”• Giving employees the tools

available and knowledge to continue work

• Combating psychological stress• Working under crisis

• Working without internet/computers• Paper timesheets• Day to day employee tasks

• Maintaining mission-critical items

• Physical transfer of items across state

Internal promotions and new employee hires

Defining construction budgets for non-federal projects going to advertisement

Approving preliminary engineering task orders

Continuing CDOT Functionality

Business aspects of Department on-hold during recovery:

• Established overall direction for the response

• Supplemented with daily priorities

Agency Collaboration

Get people paid

Cleaning computers

Prioritizing business functions

Consistent and clear messaging

Mission of the Incident Management Team

Collaborating with Partners

• Constant communication to a broad group of leadership agency wide.

• Regular touch-points throughout each day between CDOT and OIT

• Distribution of situational reports and public announcements daily

Daily Operational Rhythm

Time Meeting

8:30 - 9:00 am CDOT Exec. Management Team Call

9:00 - 9:30 am OIT Technical Call

9:30 - 10:30 am ICC Chief’s Meeting

12:00 - 12:30 pm OIT Technical Call

12:30 - 1:30 pm Regional IC Call

2:30 - 3:00 pm Divisions Meeting

3:00 - 3:30 pm OIT Technical Call

3:30 - 4:30 pm ICC Chief’s Meeting

4:30 - 5:00 pm CDOT Exec. Management Team Call

5:00 - 6:00 pm Situation Report Published

7:30 - 9:00 pm Public Announcements, DOT email

March 1st Daily Operational Rhythm

• Bringing SAP online was a high-priority to pay staff, vendors, consultants, contractors

• CDOT Staff utilized paper timesheets initially

• “SAP Labs” established so employees could share computers to enter timekeeping information

• On an average month, CDOT pays out approximately $100 Million

Get People Paid

For CDOT, SAP is an enterprise resource planning system designed to automate and integrate the majority of the Department’s business processes.

• OIT needed to check every computer, server, database, external/flash drive and application for infection:

• 3,272 laptops• 530 desktops

• Assessed backup options for infected systems:

• Cloud-based backups• Old computer drives• Public shared folders • Commvault

Cleaning Computers

Infected and Restored System Totals:• 1,274 laptops• 427 desktops • 339 servers • 158 databases• 154 software applications (like SAP)

• CDOT has thousands of business applications, hundreds of which are critical to day-to-day operations.

• Developed Excel database to continually track applications, its function, and deadline for availability

• Determined application priority, estimated availability date and responsible party

• Database was regularly distributed across regions and departments to keep staff informed of progress.

Prioritizing Business Functions

CDOT to its Staff:

• Needed consistent messaging across all Regions and Departmentsto coordinate response

• Recovery required substantial action/inaction from all employees, email less effective

• Daily calls with Regional ICCs, daily reports

• Conference “town-halls” to hear updates and ask questions

Consistent and Clear Messaging

Tracking Recovery

March 1 Variant

100% Functionality May 24

CDOT’s New Normal

Cyber attack presented opportunity to create substantial, effective system-wide changes:

• New standard security measures adopted Department-wide, almost overnight

• Revalidated CDOT firewall “safe” websites list

• Optimized CDOT OIT server demands

• Documented workarounds and messaging processes

The Open Internet(untrusted zone)

CDOT Network(trusted zone)

System Improvements

• Security improvements on all machines:

• Two-factor authentication

• Password changes

• Expedited pre-planned upgrades:• 100% laptops and tablets

• All devices on Windows 10

• Made possible with funding advance from CO Transportation Commission

PW:

=

CDOT’s New Normal

Investing in Resiliency

• Training employees • Basic computer training

• Incident Command System training (ICS 100b – 700)

• Emergency exercises

• Server Redundancy• Supplemental servers brought online

during maintenance, in event of attack

• Similar to water distribution pump system

Resiliency - The ability to adapt to changing conditions and prepare for, withstand, and rapidly recover from disruption:

Google Images

Google Images

Investing in Resiliency

• Network Segmentation• Create system in which state

networks operate independently from one another

• Reduces risk to statewide network if individual network is compromised

• Diversified Cloud Storage• Using third-party cloud services to

back up critical data

• Diversifying services used to reduce “eggs in one basket” risk

• Automating periodic data back-ups

Cycle of Life and Resiliency

Resiliency as a Mindset:

• Identifying opportunities to “build back stronger” through every step of the Recovery

• What opportunities are there to reduce risk and better protect asset in the future?

• More than just “lessons learned”, resiliency is adaptive recovery

Question