SECURITY METRICS: How to prioritize resources, demonstrate effectiveness

and enable the business

John D. Johnson

John Deere, Global Security Strategist

September 30, 2014 – IQPC IT Metrics and Performance

1

About Me

Agenda:

• How do we define security metrics?

• How are security metrics useful?

• Where do get the information, and how do we turn it into something meaningful?

• Relate it to Risk

• Relate it to Strategy

• Relate it to Maturity

• Discuss Data and Models

• Examples

• Presenting metrics to executives

• Building a security metrics program

• Q&A

3

Metrics In Real Life…

4

Measurements & Metrics

• Performance metrics measure how well an

organization performs

• Drives process improvements

• Demonstrates value-add

• Metrics can show how we compare to our peers

• We can focus our resources where they prove effective

• Management expects it • Finance did it

• Engineering did it

• HR did it

• IT did it

5

So why do we want security metrics?

• Are we being effective?

• Performance

• Controls/Processes

• Risk Management

• Are we efficient?

• Are we strategically aligned?

• Are we maturing our capabilities?

• Are we doing well compared to others?

6

The Situation

• Organizational Structure Challenges

• Budget Challenges

• Rapidly Evolving Threat Landscape

• The volume and sophistication of attacks is up

• Our future relies on our ability to take advantage

of opportunities related to social, mobile, big

data/analytics & cloud – faster, cheaper &

securely

Data-Driven Precision Farming

Big Data & Analytics

Mobility & Remote Management

Vehicle to Vehicle

Communications

Security Spend

Your Company’s

Revenue

IT

Budget

Infosec

Budget

Aligned Risk-Based Strategy

• Risk Management Framework

• Threat Intelligence

• Regulatory Knowledge

• Environmental Knowledge

• Business Knowledge

• What do we want to really protect? Focus!!!

• Meaningful Security Metrics

• Communication

Risk-Based Security Management

12

Existence of formal

RBSM function

Use of metrics to determine

RBSM effectiveness

Example RBSM Roadmap

• Understand Current State • Environment (assets (value/vulns/comp…), networks, data,

applications)

• Business knowledge (requirements, processes…)

• Regulatory environment

• Threats (std process for threat modeling/assessment)

• Capability maturity

• Determine Risk

• Prioritize Security Portfolio • Business Alignment

• Reduce risk

• Build capabilities

• Develop Metrics (tactical strategic) • Measure effectiveness at risk reduction

• Measure efficiency

• Communicate Business Value

Example: Endpoint Risk

0

0.5

1

1.5

2

2.5

3

3.5

4

0 1 2 3 4

Impact

Likelihood

Data

Loss

Advanced

Threats

Event

Logging Web

Threats

System

Attacks Endpoint

Mgmt

Basic

Malware

Meaningful Security Metrics

• Make security metrics more meaningful to stakeholders

• We need to learn to ask the right questions, if our results are going

to be meaningful

• The best metrics are SMART: Specific, Measurable, Attainable,

Repeatable & Time-Dependent

• This is an inherently difficult problem

• What is meaningful to stakeholders?

• How can we make metrics more quantitative?

• What can we measure?

• What are our peers doing?

18

• Qualitative and Quantitative Data

• Traditionally, ordinal data and storytelling has been

“good enough”

• Quantitative data can be automated & more consistent

• You probably have more data than you realize to start!

Data

Build Confidence with Metrics

• Evidence-based security strategy drives informed

decisions that have business impact

• Some metrics are better than no metrics

• Start by gathering metrics that you have

• Compliance, Exceptions…

• Security Operations, Incident Handling…

• Vulnerability Management, Code Review…

• Change Management, Configuration/Patch Management…

• Evidence may lack precision, but…

• It shows you are trying to follow a standard methodology

• It is better than FUD and hand-waiving

• It helps overcome confirmation bias

20

Example Compliance Metrics

• Manager sign-off on access controls

• A&A control artifacts

• Audit reports/findings (number, severity, BU)

• Exception reporting/tracking

• PCI Compliance status, dates

• % Linux servers at least 90% compliant with standard

21

eDiscovery Example

22

May June July August

Cases 5 9 6 11

Preservation Requests

18 33 25 40

Data Preserved 500 GB 800 GB 400 GB 300 GB

On-Site Data Collections

2 7 5 9

On-Site Data Collected

12 GB 16 GB 9 GB 40 GB

Investigations 1 collected

PST file (1200 MB)

1 USB Drive 0 2 HD Investigations

* This is an example only and all numbers are fictitious

Example Security Metrics

• Application Security • # Applications, % Critical Applications, Risk Assessment Coverage, Security Testing Coverage

• Configuration Change Management • Mean-Time to Complete Changes, % Changes w/Security Review, % Changes w/Security

Exceptions

• Financial • Infosec Budget as % of IT Budget, Infosec Budget Allocation

• Incident Management • Mean-Time to Incident Discovery, Incident Rate, % Incidents Detected by Controls, Mean-Time

Between Security Incidents, Mean-Time to Recovery

• Patch Management • Patch Policy Compliance, Patch Management Coverage, Mean-Time to Patch

• Vulnerability Management • Vulnerability Scan Coverage, % Systems w/o Known Severe Vulnerabilities, Mean-Time to

Mitigate Vulnerabilities, # Known Vulnerability Instances

23

* Source: Center for Internet Security

Metrics Measure Control Effectiveness

24

Capability Maturity

• Assess capabilities and develop roadmap to mature

BSIMM

* This is an example only and all numbers are fictitious

Security Metrics Mistakes

• Failure to track what % metric represents

• Failure to have a baseline for acceptance

• Not providing historical (consistent) trending

• Not tying metric back to business objectives

• Fear of using qualitative metrics

• Using ordinal numbers that are

ambiguous rather than

cardinal numbers

26

Models

• There are some good standard industry models and

frameworks – pick one and start

• Examples: FAIR, VERIS…

• Provide standard taxonomy for describing risk

• Standard for gathering and expressing

data in consistent manner

• Allows for analyzing complex risk

scenarios

• Be careful about how you

interpret what you see

• Correlation ≠ Causation

27

FAIR Example Risk Landscape Visibility –helps us understand how well informed (or not) our risk decisions

are. The values represent data and estimates regarding four elements (asset population, threat

conditions, value/liability at risk, and control conditions). This helps us to focus on specific areas of

poor visibility, thus improving our ability to make well-informed risk decisions. (Jack Jones)

* This is an example only and all numbers are fictitious

FAIR Example ►

* This is an example only and all numbers are fictitious

Learn Where Others Succeed & Fail

• Successful security leaders overcome confirmation bias

and compare notes more often with peers

• Standards and frameworks help a company establish a

baseline

• Results need to be translated into a context that is

relevant for your business

• Be aware that executives may downplay the significance

of industry data and feel their company is the exception to

the rule

• Choose the hill you want to die on today; you can’t boil

the ocean – FOCUS ON WHAT MATTERS

30

Presenting to Executives

31

© 2010 Institute of Operational Risk

Security Metrics for Management

• Find a way to add business value

• Meeting regulatory requirements

• Consolidation of tools, reduction of resources

• Demonstrate reduced costs by reduction in help desk cases

• Business leaders take the loss of IP seriously

• Have security seen as a business enabler. New technologies come

with risks, but they may also lead to new innovations and

competitive advantage.

• Explain it in language business leaders understand

• Make presentations clear & concise

• Avoid IT jargon

• Provide the information executives need to make informed

decisions

32

Good or Bad?

33

Good or Bad?

34

Good or Bad?

35

Applied Security Visualization, Raffael Marty

Good or Bad?

36

Clear, Concise, Contextual

37

© 2010 Institute of Operational Risk

Building a Security Metrics Program

• Decide on your goals and objectives at the onset

• Long-term and short-term goals

• Identify key metrics (SMART) to generate

• Will these be qualitative or quantitative?

• Will these be manual or automated?

• Will these be based on a standard framework, or vetted against peers, or use some

other model?

• Will these be tactical, operational, strategic or business metrics?

• Establish a baseline and targets

• Determine how best to present metrics in a consistent way, for

audience and frequency

• Get stakeholder buy-in and feedback; deliver balanced scorecard

• Develop a process for continuous improvement

38

References

• CAPEC, http://capec.mitre.org

• Verizon DBIR, http://www.verizonbusiness.com/go/2011dbir

• Verizon VERIS Framework, https://www2.icsalabs.com/veris/

• FAIR Framework, http://fairwiki.riskmanagementinsight.com/

• Center for Internet Security, Security Metrics, http://benchmarks.cisecurity.org/en-

us/?route=downloads.metrics

• Trustwave SpiderLabs Global Security Report, https://www.trustwave.com/GSR

• Ponemon Institute, http://www.ponemon.org

• Security Metrics: Replacing Fear, Uncertainty, and Doubt, Andrew Jaquith (2007)

• Metrics and Methods for Security Risk Management, Carl Young (2010)

• Security Metrics, A Beginner’s Guide, Caroline Wong (2011)

• Applied Security Visualization, Raffael Marty (2008)

• The Visual Display of Quantitative Information, Edward Tufte (2001)

39

References

• The Security Metrics Collection,

http://www.csoonline.com/article/455463/The_Security_Metrics_Collection

• Implementing a Network Security Metrics Program,

http://www.giac.org/certified_professionals/practicals/gsec/1641.php

• Choosing the Right Metric, http://www.juiceanalytics.com/writing/choosing-rightmetric/

• Web Metrics Demystified, http://www.kaushik.net/avinash/2007/12/webmetrics-

demystified.html

• Blogs about: Security Metrics, http://en.wordpress.com/tag/security-metrics/

• Standardizing metrics and their presentation,

http://www.unifiedcompliance.com/it_compliance/metrics/reporting_standards/standardi

zing_metrics_and_thei.html

• Getting to a Useful Set of Security Metrics,

http://www.cert.org/podcast/show/20080902kreitner.html

• Dashboards by Example, http://www.enterprise-dashboard.com/

• Excel Charting Tips, http://peltiertech.com/Excel/Charts/index.html

40

Q&A

John D. Johnson, Ph.D., CISSP John Deere, Global Security Strategist

JohnsonJohnD@JohnDeere.com

http://www.johndjohnson.com

http://www.linkedin.com/in/nullsession/

41