Joe Andrews, MsIA, CISSP-ISSEP, ISSAP, ISSMP, CISA Sr. Compliance Auditor – Cyber Security

Improved Compliance through Metrics and Measurement

January 30th, 2012 Mesa, Arizona

2

• Joseph A. Andrews o 21 years DoD IT Security / Network Engineering

(Federal Civilian) Information Systems Security Engineer Information Assurance Manager Network Security Engineer Information Systems Security Officer

o Academic Master of Science in Information Assurance Bachelor of Science in Information Technology Professional Certifications: CISSP-ISSEP, ISSAP,

ISSMP, CISA, CAP, GCIH, CEH, CBRM, CGEIT, CNDA

Speaker Introduction

3

• Justify the value of our activities

• Improve our ability to control and secure the infrastructure

• To better understand our Cyber Security infrastructure

Metrics: Why do we collect data?

4

• Compliance driven (e.g., NERC CIP, FISMA, ISO)

• Data retention requirements; cyber security system event logs (i.e., 90 days, 3 years for incidents, etc.)

• Collecting data that sometimes has no real context

• No real analysis of all the data collected

• Data collection activities that are not measurement driven

Reality: Data Collection Programs

5

• Metrics - Records of our

observations

Metrics and Measurement

• Measurement - The activity of

making observations and collecting data in an effort to gain practical insight into what we are attempting to understand.

6

• Records of our observations o Risk Matrix (Likelihood x Severity) o Annualized Loss Expectancy = ALE o Total Cost of Ownership = TCO o Return on Investment = ROI

Metrics: Typical Examples

7

(Basic) Risk Matrix Example

Likelihood of Event High Medium Low

High Catastrophic Bad Outlier

Medium Bad Not Good Error

Low Annoyance Typical Improbable Seve

rity

of Im

pact

8

• ALE = Annualized Loss Expectancy • ARO = Annualized Rate of Occurrence • SLE = Single Loss Expectancy

ALE = ARO x SLE

Windows Server cost $10K (system and data) with 25% chance of compromise (ARO = .25) if compromised you

expect $5K in losses (SLE = $5000)

ALE = 0.25 x $5000 = $1250

Annual Security Budget for server is $1250

Annualized Loss Expectancy

9

• Total Cost of Ownership = TCO - Hardware and Software - License and support fees - Installation and maintenance - Training - Security and Audit - Other hidden cost (i.e., utility costs)

Total Cost of Ownership

10

• Return on Investment = ROI - Expected loss for security incident is $10K - You spend $1K to prevent loss - Your ROI is now $9K - If you spend $20K to prevent the loss - Your ROI is now a negative return of 10K

Return on Investment

11

• Cyber Security metrics & statistics are in infancy stages - Industries are currently measuring limited areas of Security - Very limited industry sharing of Cyber Security data/statistics

• No common standardized central data repositories for Cyber

Security data - Computer Security Institute (CSI) and other orgs. collect data

from various industries (Computer Crime and Security Survey) - Not enough willing participants – due to security concerns –

fear of data and ultimately infrastructure compromise

• Mature industries have been collecting and sharing data for centuries (i.e., insurance, manufacturing, transportation, etc.)

Measuring opinion not actual risk

12

• Percentage of systems compliant with NERC CIP standards (CIP-005, CIP-006, CIP-007)

• Ratio of systems containing vulnerabilities as a result of a Cyber Vulnerability Assessment

• Percentage of budget devoted to compliance • Number of configuration changes or

exceptions request per time period • Average time required to remediate

vulnerabilities

Example Cyber Security Metrics

13

• Goal Question Metric method

• Security Measurement Projects

• Security Improvement Program

• Security Process Management

Security Process Mgmt. (SPM) Framework

14

15

• Invented in 1970, by Victor Basili and was originally developed for NASA: software engineering practices; aligning software metrics with software goals o 1. Define goals and objectives the measurement is

supposed to achieve

o 2. Translate goals into specific questions that must be answered

o 3. The questions are answered by identifying and developing the appropriate metrics

Goal Question Metric method Three step process

16

GQM diagram

17

• Specific o Bad: We’re going to improve Cyber Security o Good: We’re going to reduce response times to Cyber

Security incidents by 10% • Defined boundaries and attributes

o business unit, system, concept • Attainable • Verifiable

** Note: Keeping goals too vague diminishes

your accomplishments

(Good) Goal attributes

18

• Goal Components - Outcome: Increase - Element: Enforcement of Cyber Security Policy - Element: User Awareness of the Policy - Element: User Acknowledgement of the Policy - Perspective: Compliance manager • Goal statement The goal of this project is to increase the enforcement and user awareness of the Cyber Security policy, by increasing user acknowledgements of the company’s security policy documents from the perspective of the compliance manager.

GQM – Enforcement of Security Policy - Example 1 CIP-3, R1 CIP-7, R5.2

19

• Question 1: What is the current level of enforcement of the Cyber Security policy?

• Metric: Number of reported security policy violations in the previous 12 months

• Metric: Number of enforcement actions taken against policy violations in previous 12 months

GQM – Enforcement of Security Policy - Example 1 CIP-3, R1 CIP-7, R5.2

20

• Question 2: What is the current structure of the Cyber Security policy?

• Metrics: - Number documents that make up the security

policy - Format(s) of security policy documents - Location(s) of security policy documents - Types of acknowledgement mechanisms - Length of time since last review by mgmt.

GQM – Enforcement of Security Policy - Example 1 CIP-3, R1 CIP-7, R5.2

21

• Goal Components - Outcome: Understand, observe, elicit, improve - Element: Physical security practices and behaviors - Element: Employee explanations and opinions - Element: Physical and Cyber Security posture - Perspective: Compliance, Physical and Cyber Security teams • Goal statement The goal of this project is to understand the physical security practices and behaviors within the company by observing physical activities and eliciting employee explanations and opinions regarding these activities, in order to improve the company’s physical and Cyber Security posture from the perspective of the Compliance, Physical and Cyber Security teams

GQM – Tailgating - Example 2 CIP-6, R5

22

• Question 1: What are the physical security practices and behaviors taking place throughout the company?

• Metric: Ethnographic/human observation of company facilities (entryways) and employee activities (e.g., tailgating, facilitating tailgating)

• Question 2: Why are physical security practices and behaviors undertaken?

• Metric: Observations, interviews, and discussions with employees and other stakeholders of the company

GQM – Tailgating - Example 2

Ethnography = qualitative research involving the observation in behavior of groups and/or societies

CIP-6, R5

23

• Question 3: How is physical security perceived and enacted by the members of the company?

• Metric: Qualitative analysis of the data gathered (interviews, observations) to identify categories, patterns, and themes regarding the practices of physical security

GQM – Tailgating - Example 2 CIP-6, R5

24

• Tailgating narratives identified during project o Tailgating is understandable

- Culture of Trust (Company fosters trusting environment) - Avoiding confrontation - Matter of convenience o Tailgating must be prevented

- Theft, and other loss potential (acknowledged) - Keeping people safe - Physical access may allow “hackers” to potentially compromise systems o Tailgating is hard to prevent

- Too expensive (badge readers, cameras, guards) - Lack of compatibility (doors, badge readers, difficult to centrally manage) - Physical locations encourage tailgating

GQM – Tailgating - Example 2 CIP-6, R5

25

Tailgating diagram

26

• Goal Components - Outcome: Improve, assess - Element: Compliance rates - Element: Readability and difficulty - Element: Security policy documents - Perspective: Security policy user • Goal statement The goal of this project is to improve the security policy compliance rates, by assessing the readability and difficulty levels of different policy documents from the perspective of the general security policy user

GQM – Policy Readability - Example 3 CIP-002 - 009

27

• Question 1: How difficult is it to read and understand company security policy documents?

• Metric: Readability software & test standards (e.g., Readability Studio & Flesch Readability Ease Test; sentence length and # of syllables)

• Question 2: Are the readability levels for the security policy documents appropriate for the specific policy user audience?

• Metric: Estimated reading levels for policy document users (based on known education levels)

GQM – Policy Readability - Example 3 CIP-002 - 009

28

Statistics Data ------------------------------------------------------------- Number of sentences = 110 Number of difficult sentences = 55 (50%) Average sentence length = 23.5 words Minimum grade level (suitable) = 16+ graduate

level education

GQM – Policy Readability - Example 3 CIP-002 - 009

29

GQM – Policy Readability - Example 3

0 500 1000 1500 2000 2500

Total Words

6+ Character Words

3+ Syllable Words

Monosyllabic Words

2290

1085

700

875

Total Words

Readability software results

**Online tools are available

CIP-002 - 009

30

Score Description 90-100 Very Easy 80-89 Easy 70-79 Fairly Easy 60-69 Standard 50-59 Fairly Difficult 30-49 Difficult 0-29 Very Confusing

GQM – Policy Readability - Example 3

Flesch Readability Ease Test Scores

CIP-002 - 009

31

• Potential liabilities – Complex Readability o Security breaches by employees who

couldn’t understand the policy o Potential lawsuits for wrongful

termination of employees fired for policy violations

GQM – Policy Readability - Example 3 CIP-002 - 009

32

Don’t dumb down policies Too Much! BEWARE: Special & Mail Room Employees!

33

Don’t dumb down policies too much!

34

Don’t dumb down policies too much!

35

36

Establish Security Metrics Catalog Securi ty Metr ics Catalog

Goals and Projects # Associated Metr ics

Perimeter Securi ty

1 CVA Results: (# or %) vulnerable Access Points

2 # or % of Access Points with TFEs

3 Ports and Services: % Compliant Access Points

Endpoint Securi ty

1 CVA results: Vulnerable CCAs (# or %)

2 # or % of Cyber Assets with TFEs

3 Ports and Services: % Compliant Access Points

Securi ty Pol icy

1 Time between Cyber Security Policy reviews

2 Readability of Cyber Security Policy

3 # of Cyber Security policy violations (prev. 6 mo.)

37

• Logistics and organizational structure o Decide project type (descriptive,

experimental, compliance) o Conduct Goal-Question-Metric analysis o Conduct a review for previous efforts o Consider data sources & analysis requirements o Get buy in from management and

stakeholders

Security Measurement Project (SMP)

38

• Interconnects tactical Security Measurement Projects over time

• Measuring security operations becomes a strategic effort (tactical to strategic)

• Forms a knowledge loop • Defining, managing and improving

collaborations between projects • Making informed decisions that improve

Cyber Security

Security Improvement Program (SIP)

39

SIP Document Number: SIP2013.01-30 General Project Data: Completed Projects = 3 Active Projects = 1 Proposed Projects = 1 Security Measurement Project A Project Name / Number: Policy Readability Assessment SMP2013.01 Project Sponsor / Lead: Sponsor - B. Castagnetto – CIO Lead - J. Andrews – CISO Project Begin / End: Begin: 11.04.12 End: 01.30.13 SMP GQM Goal(s): Assess the readability of the corporate security

policy from the perspective of the policy user Questions: … Metrics: …

SIP document structure

40

SIP Diagram

41

SIP Diagram

DATA > INFORMATION >> KNOWLEDGE >>> WISDOM

42

43

1st CVA Project

2nd CVA Project w/ new vulnerability remediation time concerns

3rd CVA Project

1st Budget Project 2nd Budget Project

44

• Identify stakeholders and sponsors • Goal, questions and metrics analysis • Project cost and project benefits • Risk analysis results (be upfront) • Formal acceptance (formal signoff by

stakeholders

Business Case for Metrics

45

• Deconstruct all the activities that make up your Cyber Security Business Process o Who owns the process? o Who completes each process? o What systems are involved with each

activity? o How much does each activity cost? o How long does each activity take?

Cyber Security = Business Process

46

47

• Compliance is a component of Cyber Security

• The Cyber Security infrastructure is the foundation for compliance

• By only focusing on compliance, you are engaging in narrow-focused checkbox security

Compliance & Cyber (Security)

48

• By applying context and experience, metrics data can be transformed into Corporate Wisdom o Data (no context) o Information o Knowledge o Wisdom (Corporate Wisdom)

DIKW Hierarchy

49

DIKW diagram

50

You can not manage what you can not measure, and you can not

measure what you do not understand.

Metrics and Measurement

51

52

• Hayden, L., IT Security Metrics: A Practical Framework for Measuring Security & Protecting Data. McGraw Hill, 2010

• 2008 Annual Report: IT Governance, Risk, and Compliance – Improving Business Results and Mitigating Financial Risk. IT Policy Compliance Group, 2008, from www.itpolicycompliance.com/research reports

• Jaquith, A., Security Metrics: Replacing Fear, Uncertainty and Doubt. Addison-Wesley, 2007. • Hubbard, D., How to measure Anything: Finding the Value of Intangibles in Business. Wiley, 2007. • Goodwin, P., Decision Analysis for Management Judgment. Wiley, 2004 • Jacka, J. Mike, and P. Keller. Business Process Mapping: Improving Customer Satisfaction. Wiley,

2002. • The Hulk photo: The Hulk yelling (2012). Retrieved from Fanpop website on January 15, 2013, from,

http://www.fanpop.com/clubs/dr-bruce-banner/images/31267409/title/hulk-yelling-photo • Werewolf photo: Top ten scariest Halloween monsters (2011). Retrieved from Ten-O-Rama website

on January 16, 2013, from, http://www.tenorama.com/en/ranking/top-10-scariest-halloween-monsters • Ringu photo: 31 Days of Horror, The October Movie Marathon (2012). Retrieved from NeoGaf

Believe website on January, 19, 2013, from, http://www.neogaf.com/forum/showthread.php?p=42427616

References

Joe Andrews, CISSP-ISSEP, ISSAP, ISSMP, CISA Sr. Compliance Auditor – Cyber Security Western Electricity Coordinating Council jandrews[@]wecc[.]biz Office: 801.819.7683

Questions?