DeepSec 2014 - The Measured CSO

THE MEASURED CSOALEX  HUTTON  -­‐  A  TOO  BIG  TO  FAIL  BANK  

@ALEXHUTTON

SECTION 1: BACKGROUND Who am I? What is this topic? Where are we? How did we get here?

SECTION 2: ON THE ROLE OF THE CSO What is a CSO? What do they do? What is success? How do they get there?

SECTION 3: BECOMING MEASURED What does that mean? What do we need? How do we do it? Where shall we go?

SECTION 1: BACKGROUND Who am I? What is this topic? Where are we? How did we get here?

SECTION 2: ON THE ROLE OF THE CSO What is a CSO? What do they do? What is success? How do they get there?

SECTION 3: BECOMING MEASURED What does that mean? What do we need? How do we do it? Where shall we go?

SECTION 1: BACKGROUND Who am I? What is this topic? Where are we? How did we get here?

SECTION 2: ON THE ROLE OF THE CSO What is a CSO? What do they do? What is success? How do they get there?

SECTION 3: BECOMING MEASURED What does that mean? What do we need? How do we do it? Where shall we go?

SECTION 1: BACKGROUND Who am I? What is this topic? Where are we? How did we get here?

1.1 WHO AM I

• Security Engineer

• Security Product Management

• E-Commerce Site Design / Manager

• Risk Consultant

• OCTAVE / NIST

• FAIR

• Verizon DBIR

• IANS Faculty

• Director, Operations / Technology Risk

• Director, Information Security

1.1 WHO AM I

1.2 WHAT IS THIS TOPIC

“…when you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced to the stage of science, whatever the matter may be.”

William Thomson, 1st Baron Kelvin & Measurement Badass

The Journey Towards Knowledge (and therefore, security)

1.2 WHAT IS THIS TOPIC

WHERE ARE WE (OUR INDUSTRY)

Security is now so essential a concern that

we can no longer use adjectives and adverbs

but must instead use numbers.

Dan Geer, Security Badass

Unfortunately…

Science is based on inductive observations to derive meaning and understanding and measurement on quality (ratio) scales, so what about InfoSec?

Where do we sit in the family of sciences?

We’re the Crazy Uncle with tinfoil hat antennae used to talk to the space aliens of Regulus V, has 47 cats, and who too frequently (but benignly) forgets to wear pants.

Take, for example, CVSS

“the Base Equation multiplies Impact by 0.6 and Exploitability by 0.4”

= ShinyJet Engine X Peanut Butter

“the Base Equation multiplies Impact by 0.6 and Exploitability by 0.4”

20

adding onewilly-nilly doesn’t suddenly transformordinal rankings into ratio values.

decimals aren’t magic.

At our present skill in measurement of security, we

generally have an ordinal scale at best, not an interval scale

and certainly not a ratio scale. In plain terms, this means we

can say whether X is better than Y but how much better and

compared to what is not so easy.

– Again, Baddss Dan Geer

State of the Industry- proto-science - somewhat random fact

gathering (mainly of readily accessible data)

- a“morass”of interesting, trivial, irrelevant observations

- a variety of theories (that are spawned from what he calls philosophical speculation) that provide little guidance to data gatheringThomas Kuhn Philosophy of Science Badass

1.3 HOW DID WE GET HERE

1.3 HOW DID WE GET HERE

The tragedy of two mistakes

FIRST MISTAKE: LIMITING OURSELVES(security is an engineering issue?)

• OSI Model (original version)

• OSI Model (SOA Remix)

• OSI Model (Mika’s 12” Extended Dance Version)

10: Religion Operator Layer

SECOND MISTAKE: BLIND LEADING THE BLIND

BLIND MAN 1: THE FUD FACTORY

FUD FACTORY EXAMPLE - MOBILE VS WEB

Google Trend: Web Security Mobile Malware

#RSAC

36

Clustering of over 5,000 incidents

Espionage

Point of Sale

Skimming Devices

Theft/Loss

Error

EmployeeMisuse

Web Applications

DBIR Top Patterns:

Web Only:

Web Applications

In FinServ vs. All Industries

DBIR Global Representation of Assets in Cases:

DBIR Global Representation of Assets in Cases:

NHTCU investigation into groups using mobile malware showed that in less than a year’s time, five variations of mobile malware for one specific bank could be detected. Modest estimates suggest that criminals gained around €50,000 per week using this specific form of mobile malware, harvesting over 4,000 user credentials from 8,500 infected bank customers in just a few months. Mobile malware does not move the needle in our stats as we focus on organizational security incidents as opposed to consumer device compromises.

DBIR Global Representation of Assets in Cases:

NHTCU investigation into groups using mobile malware showed that in less than a year’s time, five variations of mobile malware for one specific bank could be detected. Modest estimates suggest that criminals gained around €50,000 per week using this specific form of mobile malware, harvesting over 4,000 user credentials from 8,500 infected bank customers in just a few months. Mobile malware does not move the needle in our stats as we focus on organizational security incidents as opposed to consumer device compromises.

BLIND MAN 2: THE ACCOUNTING-CONSULTANCY INDUSTRIAL COMPLEX

Complex (adaptive)Systemsa system composed of interconnected parts that as a whole exhibit one or more properties not obvious from the properties of the individual parts

These “risk” statements you’re making...

I don’t think you’re doing it right.

- (Chillin’ Friederich Hayek)

BLIND MAN 3: OUR BROKEN MODELS

“the Base Equation multiplies Impact by 0.6 and Exploitability by 0.4”

ROYTMAN: ON VULNERABILITIES

ROYTMAN: ON VULNERABILITIES

A CSO MUST BECOME “MEASURED” TO ESCAPE THE MISTAKES OF THE PAST AND PUSH INTO THE FUTURE

SECTION 2: ON THE ROLE OF THE CSO What is a CSO? What do they do? What is success? How do they get there?

• What Is a CISO (throne of blood image

WHAT IS A CSO

• What Is a CISO (throne of blood image

WHAT IS A MEASURED CSO

W.E. DEMING

Father of Total Quality Management and inspiration that drove the Japanese “post-war economic miracle.”

IT WAS NO MIRACLE. What Deming taught the Japanese was “management by fact.”

• Improvements to the system are never ending.

• The only people who really know where the real potentials for improvement are the workers.

• The system is always changing.

• There are countless ways for the system to go wrong.

• Statistics (metrics) are used to focus the conversation on fact and improvement

• Goals for quality are cross-silo

• Theories for improvements are implemented and tested.

• The management uses the workers as essential "instruments" in understanding what is.

A MEASURED CSO:

• Relies on metrics, data, intel for good decisions,

• Invests in improvements to People, Process and Technology,

• Puts innovation for improvements to the system (improvements = security, cost) in the hands of the operator,

• Ensures that there is a feedback loop for effectiveness initiatives, and

• Works tirelessly within the bureaucracy to improve all aspects of the system.

THE MEASURED CSO’S MISSION:

• To provide the best and least-cost security for shareholders, and continuity of employment for his workers.

• We, as an industry, know that “best” and”least-cost” are not necessarily contradictory

• We also have a HUGE continuity issue

THE MEASURED CSO USES METRICS TO IMPROVE THE SYSTEM.

WHAT IS THAT SYSTEM - That which Defends (Detects, Responds, & Prevents).

THE MEASURED CSO USES METRICS TO:

• Develop and improve the People, Process, and Technology to Defend

• Plan / Build / Manage those defenses

THE SYSTEMS FOR DEVELOPING METRICS ARE MORE IMPORTANT THAN THE SYSTEMS OF DOGMA THAT DEFINE “STANDARDS” OF OPERATION.

THE SYSTEMS FOR DEVELOPING METRICS ARE MORE IMPORTANT THAN THE SYSTEMS OF DOGMA THAT DEFINE “STANDARDS” OF OPERATION.

Sorry, ISACA

THE SYSTEMS FOR DEVELOPING METRICS ARE MORE IMPORTANT THAN THE SYSTEMS OF DOGMA THAT DEFINE “STANDARDS” OF OPERATION.

• There are two systems which the CSO must manage across (at least 4 audiences)

• Those that support “defend”

• Those that support Plan/Build/Manage

MEASURED CSO SYSTEM 1: THE METRICS AND MODELS THAT “DEFEND”

EPIDEMIOLOGY

EPIDEMIOLOGYRisk Factors (Determinants) Variables associated with increased frequency of event.

Risk Markers Variable that is quantitatively associated with a disease or other outcome, but direct alteration of the risk marker does not necessarily alter the risk of the outcome.

Correlation vs. Causation Risk factors or determinants are correlational and not necessarily causal, because correlation does not prove causation.

EPIDEMIOLOGYRisk Factors (Determinants) Variables associated with increased frequency of event.

Risk Markers Variable that is quantitatively associated with a disease or other outcome, but direct alteration of the risk marker does not necessarily alter the risk of the outcome.

Correlation vs. Causation -Risk factors or determinants are correlational and not necessarily causal, because correlation does not prove causation.

THE MEANS TO FIND PATTERNS

Example of a medical approach:Dr. Peter Tippett & Verizon DBIR

A security incident (or threat scenario) is modeled as a series of events. Every event is comprised of the following 4 A’s:

Agent: Whose actions affected the asset Action: What actions affected the asset Asset: Which assets were affected Attribute: How the asset was affected

VERIS (Vocabulary for Event Recording & Incident Sharing)

70

72

Object-Oriented Modeling

VERIS (Vocabulary for Event Recording & Incident Sharing)

73

1 2 3 4 5 >" >" >" >"Incident as a chain of events >"

Object-Oriented Modeling

VERIS (Vocabulary for Event Recording & Incident Sharing)

74

1 2 3 4 5 >" >" >" >"Incident as a chain of events >"

A “Pattern”

VERIS: Classification of Events by Risk Factor

Complex System?

VERIS FOUND PATTERNS!

#RSAC

36

Clustering of over 5,000 incidents

Espionage

Point of Sale

Skimming Devices

Theft/Loss

Error

EmployeeMisuse

Web Applications

DBIR Top Patterns:

THE KEY TO THE MEASURED CSO SYSTEM 1: FRAMEWORK, DATA, MODELS

√∫∑

Framework

Models Data=

∩

VERIS+

actor information

asset information

impact information

controls information

risk

Classifying sets of security information

√∫∑

Framework

Models Data=

∩Data Warehousing+

82

Apache Storm

83

Data MapReduce Process Analytics & Reporting

Threat Intel FeedsControl DataControl LogsSystem Logs

Event  History  &  Loss  Loss  Distribu8on  Dev.  B.I.A.

Control DataControl LogsSystem Logs

Configuration DataVulnerability DataHR InformationProcess Behaviors

XMLCSVEDI

LOGSQL

JSONText

BinaryObjects

create map

reduce

TraditionalRDBMSSystems

Workflow

Analytics

Reporting

Models suggesting IO

C= true

88

1 2 3 4 5 >" >" >" >"Incident as a chain of events >"

89

1 2 3 4 5 >" >" >" >"Incident as a chain of events >" X X X

90

Example of data enrichment:

Asset Intel : Vendor-owned SaaS application

√∫∑

Framework

Models Data=

∩

MEASURED CSO SYSTEM 1: THE METRICS AND MODELS THAT “DEFEND” AGAINST THREAT PATTERNS.

(real and anticipated or forecasted)

MEASURED CSO SYSTEM 2: THE METRICS NEEDED TO PLAN/BUILD/MANAGE SYSTEMS (OPERATIONS)

THE MEASURED CSO MUST ALSO INCLUDE A KEEN UNDERSTANDING AND PARTNERSHIP WITH IT OPERATIONS

THE MICROMORT A one in a million chance of death Ronald A. Howard

Activities that increase the death risk by roughly one micromort, and their associated cause of death (wikipedia):

Traveling 6 miles by motorbike (accident) Traveling 17 miles by walking (accident) Traveling 10 miles by bicycle (accident) Traveling 230 miles (370 km) by car (accident) Traveling 1000 miles (1600 km) by jet (accident) Traveling 6000 miles (9656 km) by train (accident) Traveling 12,000 miles (19,000 km) by jet in the United States (terrorism) Increase in death risk for other activities on a per event basis:

Hang gliding – 8 micromorts per trip Ecstacy (MDMA) – 0.5 micromorts per tablet (most cases involve other drugs)

Modern Risk Management is not only bad at describing risk, but it also is focused on reporting its own version “micromorts”Inefficiently.

Traveling 10 miles by bicycle (accident)

Modern Risk Management is not only bad at describing risk, but it also is focused on reporting its own version “micromorts”Inefficiently.

Traveling 10 miles by bicycle (accident)

Ecstacy (MDMA) – 0.5 micromorts per tablet (most cases involve other drugs)

Modern Risk Management is not only bad at describing risk, but it also is focused on reporting its own version “micromorts”Inefficiently.

Traveling 10 miles by bicycle (accident)

Modern Risk Management is not only bad at describing risk, but it also is focused on reporting its own version “micromorts”Inefficiently.

Traveling 10 miles by bicycle (accident)

The Measured CSO must know where IT is overweight, smoking ecstasy, while riding a rocket-powered bicycle on the railing of a bridge.

DATA: VISIBLE OPS FOR SECURITY

104

Example of data enrichment:

Asset Intel : Vendor-owned SaaS application

SECTION 3: BECOMING MEASURED What does that mean? What do we need? How do we do it? Where shall we go?

106

MOST METRICS PROGRAMS

If we consider a single metric as a building block

108

It should be used by the CSO to paint a picture of the security program

109

Whose context is the whole of IT.

110

But because we gather what is most readily available - most metrics programs look like my living room.

How does the measured CSO get context?

GOAL, QUESTION, METRICConceptual level (goal) goals defined for an object for a variety of reasons, with respect to various models, from various points of view.

Operational level (question)

questions are used to define models of the object of study and then focuses on that object to characterize the assessment or achievement of a specific goal.

Quantitative level (metric)

metrics, based on the models, is associated with every question in order to answer it in a measurable way.

Victor Basili

GQM FOR FUN & PROFIT

Goals establishwhat we want to accomplish.

Questions help us understand how to meet the goal. They address context.

Metrics identify the measurements that are needed to answer the questions.

Goal 1 Goal 2

Q1 Q2 Q3 Q4 Q5

M1 M2 M3 M4 M5 M6 M7

Execution

Models

Data

Goal 1 Goal 2

Q1 Q2 Q3 Q4 Q5

M1 M2 M3 M4 M5 M6 M7

GQM FOR FUN & PROFIT

GQM EXAMPLE: PATCH MANAGEMENT

Patching Scorecard

Goal 1: Comprehensive

Goal 2: Timely

Goal 3: Cost Efficient

GQM EXAMPLE: PATCH MANAGEMENT

Patching Scorecard

Goal 1: Comprehensive

Goal 2: Timely

Goal 3: Cost Efficient

% Coverage by Business Units

%Coverage by Asset category

%Coverage by Risk

Unix

Windows Server

DesktopOS

Components

Likelihood

Impact

Most Significant Failures

Repeat Offenders

By Asset Category

By Location (DMZ, Semi-Pub, Internal)

By Business Unit

By Asset Category

By Location (DMZ, Semi-Pub, Internal)

By Business Unit

GQM EXAMPLE: PATCH MANAGEMENT

Patching Scorecard

Goal 1: Comprehensive

Goal 2: Timely

Goal 3: Cost Efficient

What should our Priorities be for timeliness?

What is Policy for timeliness?

What other Considerations for Timeliness?

What is time to patch like for assets with worst Likelihoods?

What is time to patch like for assets with worst Impacts?

What % are Late by

What are our Repeat Offenders?

likelihood

Impact

by asset category

by business unit

by risk

UNIX

Windows Server

Desktop

likelihood

impact

GQM EXAMPLE: PATCH MANAGEMENT

Patching Scorecard

Goal 1: Comprehensive

Goal 2: Timely

Goal 3: Cost Efficient

Cost

Risk Reduction

Hour per Asset spent PatchingBy Asset Category

By Location (DMZ, Semi-Pub, Internal)

By Cost Per Hour

Hour per Asset, by ALE per Hour

Hour per asset category

GQM EXAMPLE: PATCH MANAGEMENT

• The Measured CSO creates a scorecard of KRI’s & KPI’s that Includes:

• Historical values

• “Triggers”

• “Thresholds”

(each of these?) aren’t perfect, but establish a hypothesis for testing & optimization.

Now you’re ready to come correct, my Bias!

- (Chillin’ Friederich Hayek)

MEASURED CSO FRAMEWORK FOR GQM: NIST CSF

NIST CSF

Identify

Protect

Detect

Respond

Recover

Asset Management

Business Environment

risk assessment

risk management strategy

Governance

Access Control

Awareness and Traininig

Data Security

Information Protection Processes and Procedures

Maintenance

Protective Technology

Anomalies and Events

Security Continuous Monitoring

Detection Processes

Response Planning

Response Communications

Response Analysis

Response Mitigation

Response Improvements

Recovery Planning

Improvements

Communications

SECTION 3: BECOMING MEASURED What does that mean? What do we need? How do we do it? Where shall we go?

√∫∑

Framework

Models Data=

∩

124

Example of data enrichment:

Asset Intel : Vendor-owned SaaS application

ETL AND STORE ALL THE THINGS!!!

126

Data MapReduce Process Analytics & Reporting

Threat Intel FeedsControl DataControl LogsSystem Logs

Event  History  &  Loss  Loss  Distribu8on  Dev.  B.I.A.

Control DataControl LogsSystem Logs

Configuration DataVulnerability DataHR InformationProcess Behaviors

XMLCSVEDI

LOGSQL

JSONText

BinaryObjects

create map

reduce

TraditionalRDBMSSystems

Workflow

Analytics

Reporting

Models suggesting IO

C= true

“If you do not know how to ask the right question, you discover nothing.”

RESOURCESFOR  GQM  AND  MICROMORTS  -­‐  WIKIPEDIA  FOR  DBIR  DATA,  THE  VERIZON  DBIR  FOR  DEMING  QUOTES,  THE  WORKS  OF  MYRON  TRIBUS:  

http://www.qla.com.au/papersTribus/Oslo3.pdf  http://www.unreasonable-­‐learners.com/wp-­‐content/uploads/2011/03/Germ-­‐Theory-­‐of-­‐Management-­‐Myron-­‐Tribus1.pdf  

http://www.qla.com.au/papersTribus/DEMINGS_.PDF