SEM01A-Using Information Security Metrics to Demonstrate Value and Drive Improvements (separate registration required) (236663454)

8/11/2019 SEM01A-Using Information Security Metrics to Demonstrate Value and Drive Improvements (separate registration required) (236663454)

http://slidepdf.com/reader/full/sem01a-using-information-security-metrics-to-demonstrate-value-and-drive-improvements 1/100

E D U C A U S E S E C U R I T Y P R O F E S S I O N A L S C O N F E R E N C EM AY 6 - 8 , 2 0 1 4

Using Information SecurityMetrics To Demonstrate Valueand Drive Improvements

SHIRLEYC. PAYNE

AVP FOR INFORMATION SECURITY, POLICY, & RECORDS

UNIVERSITY OFVIRGINIA



Copyright Shirley C. Payne 2014.

This presentation leaves copyright of the content to the presenter.Unless otherwise noted in the materials, uploaded content carries theCreative Commons Attribution-NonCommercial-ShareAlike license,

which grants usage to the general public with the stipulated criteria.



Seminar Will Answer…

DemonstrateValue & DriveImprovements

What makesa metriceffective?

What are thechallenges?

Where do Istart?

How should I

communicatemetrics?

Where can Ilearn more?

a nd provide lots of examples…



DemonstrateValue & Drive

Improvements

What makes a

metric effective?



Provide single-point-in-

time views of specific,discrete factors

Generated by counting

Objective raw data

Derived by comparing

two+ measurementstaken over time to apredetermined baseline

Generated by analysis

Objective or subjectivehuman interpretations ofthose data

Measurements and Metrics – same thing?



The Mark of Good Metrics

Metrics should be SMART

Specific Well-defined, using unambiguous wording

Measurable Quantitative when feasible

A ttainable Within budgetary and technical limitations

Repeatable Measurements from which metric is derived donot vary depending on the person taking them.

T ime-dependent Takes into consideration measurements frommultiple time slices

George Jelen, “SSE -CMM Security Metrics”



Albert Einstein

Everything thatcan be counted

does notnecessarily count;

everything that

counts cannotnecessarily becounted.



Truly Effective Metrics…

Indicate the degree to whichsecurity goals are being met

Show linkage between securityand institutional goals

Drive actions taken to improvethe overall security program



Rate This Metric

% of servers that are secure has increased fourfold since 2010

S

M

A

R

T



Rate This Metric

% of servers that are secure has increased fourfold since 2010

% of servers with patched operating systemsincreased fourfold since 2010



Rate This Metric

% of employees who are aware of security threats doubled lastyear

S

M

A

R

T



Rate This Metric

% of employees who are aware of security threats doubled lastyear

% of employees completing annual securityawareness training doubled last year



Rate This Metric

Level of faculty frustration w/2-factor authentication comparedto reduced risk of unauthorized data access



Rate This Metric

Level of faculty frustration w/2-factor authentication comparedto reduced risk of unauthorized data access

% of faculty issued UVa’s hardware identity tokenscompared to faculty use of tokens for email login



Rate This Metric

Web application vulnerabilities found during January 2014penetration test

S

M

A

R

T









Rate This Metric

In 2013 there were 98 reported Higher Education breachesnationwide compared to 5 at this institution.

S

M

A

R

T



Rate This Metric

In 2013 there were 98 reported Higher Education breachesnationwide compared to 5 at this institution.

In 2013 there were 5 reported breaches, 4 of which were discovered by internal controls (versus

reported by outsiders)



The Value of “Truly Effective” Security Metrics(internal focus)

Discern effectiveness of particular security programcomponent

Indicate security of specific system, product, or process

Identify risk in not taking a given action and, thereby, helpprioritize corrective actions

Provide evidence of regulatory compliance

Demonstrate ability of security staff and departments toaddress security issues for which they are responsible



Value of “Truly Effective” Security Metrics(external focus)

Provide basis for answering tough questions, such as Are we more secure today than we were before?How do we compare to others in this regard?

Are we secure enough?

Raise security awareness among executives and otherstakeholders

Clearly convey value of overall security program relative tobusiness objectives



What makes a metric effective?

Characteristics ofeffective metrics:

SMARTIndicate % to whichsecurity goals are met

Link security toinstitutional goals

Drive improvements

EffectiveMetric

Specific

Measurable

AttainableRepeatable

Time-dependent



What makes a metric effective? A metric is effective if it can:

Provide insight into IS program effectiveness,regulatory compliance, and ability to address securityconcerns

Help identify risks of not taking certain actions,providing guidance for future investments.

Provide concrete facts for raising security awareness

Provide credible answers to hard questions aboutstatus and value of IS program




Improvements

What are thechallenges?




The State of Security Metrics

Other disciplines, such as the field of finance, have provenquantitative methods for determining risk, along withdecision-making frameworks based on establishedmeasures and metrics.

[These] are just emerging for information security, however,and as in any discipline, require realistic assumptions andinputs to attain reliable results.

Wayne Jansen, “Directions in Security Metrics Research,” NISTIR 7564; April 2009



75%CISOs say…





51%CISOs say their…



WhyNot?

%



18

35

40

48

59

0 10 20 30 40 50 60 70

EXECS NOT INTERESTED

TIME/RESOURCES TO PREP REPORTS FOR EXECS

ONLY COMMUNICATE W/ EXECS ON INCIDENTS

HIGHER PRIORITIES

INFO TOO TECHNICAL FOR EXECS

%

%





Let’s Look At The Challenges

Measuring Risk

Determining ROSI

Limited Guidance and Practical Examples



How To Measure Risk?

Risk = Asset Value x Threat x Vulnerability

Asset Value – easiest to measure in some cases, buthow to quantify assets like institutional reputation?

Threat – very hard to measure the potential for harm,although information from external sources may be

useful.Vulnerability – sources of good information available,but not all vulnerabilities can be quantified.



Determining ROSI?

“It’s a good idea in theory, but it’smostly bunk in practice… Securityis not an investment that providesa return… It is an expense that,hopefully, pays for itself in costsavings…Security is about lossprevention, not about earnings. “

Bruce Schneier – September 2, 2008https://www.schneier.com/blog/srchives/2008/09/security_roi_1.html



Guatemala Sinkholehttp://news.nationalgeographic.com



“We’re here to suggest not only

that you can use ROSI to sell

security internally, but you must.”

Scott Berinato, “Calculated Risk: Return on Security Investment,www.csoonline.com

An Alternate View!



Challenge industry assumptions and culturalbiases

Rethink YourAssumptions

Find and use data that’s out there

Do theLegwork

Subtract cost from benefits

Do the Math

Scott Berinato, “Calculated Risk: Return on Security Investment, www.csoonline.com



Challenge industry assumptions andcultural biases

Rethink YourAssumptions

• Precision is not the goal• Think in stochastic, not binary, terms

Fire extinguisher ROI: $3 return for every $1 investedNOT

Fire extinguisher ROI: $3.14 return for every $2.97 invested



Find and use data that’s out there

Do theLegwork

• Actuarial information, e.g., CERT, Poneman• Annual data breach reports, e.g., Verizon, privacyrights.org• Threat trends, e.g., IBM X-Force, Mandiant• Talk to business managers, e.g., Risk Management Officers,

Financial Managers




Do the Math

• Annual Loss Expectancy• Modified ALE• Other methods



Cost Examples

Lost staff productivityLoss/compromise of dataRecovery costs

Reputational lossFines and lawsuitsLoss of future research grants/contracts

Etc.

Informed by Julia Allen – March 10, 2003 “Making the BusinessCase for Information Security: Selling to Senior Managements”



Scenario: Need to determine ROSI on acquisition ofweb app vulnerability scanning service




Do the Math

• Annual Loss Expectancy: how much $$ lost per year due tosecurity incident

ALE =average cost of data breach X probability of web app breach next year

A L E = $3.2M X .22 = $704,000

Poneman Study Verizon Study






Do the Math


• Modified ALE: ALE w/effect of mitigation measure incorporated

COST SAVINGS = ALE – mALE

COST SAVINGS = $704,000 - $352,000 = $352,000




Do the Math


• Modified ALE: ALE w/effect of mitigation measure incorporated

ROSI = BENEFITS - COST

Cost per year of xyz web app service = $80,000

ROSI = $352,000 - $80,000 = $281,600



Let’s Look At The Challenges

Limited Guidance and Practical Examples



Limited Guidance and Practical Examples?Good News!

ISO 27004

NIST SP 800-55 Rev. 1

CIS Consensus Information Security MetricsTop 20 Critical Security Controls



ISO/IEC 27004

Published December 2009 (new version planned)Guidance for developing metrics for evaluatinginformation security programsKey sections:

Information security measurement overview;Management responsibilities;Measures and measurement development;Measurement operation;

Data analysis and measurement results reporting;Program evaluation and improvement.

http://www.iso.org/iso/catalogue_detail?csnumber=42106





NIST SP 800-55 Rev. 1

Published July 2008Specific advice for developing, selecting, and implementingperformance measures

Security controls tied to overall missionPractical examples

http://csrc.nist.gov/publications/PubsSPs.html





CIS Consensus Information Security Metrics

V1.1.0 published November 2010Metrics on security outcomes and process performance.

Common definitions for data collection and analysis

Metrics grouped by purpose and audience: management,operational, technical

Twenty metrics defined in six functions: incident management,vulnerability management, patch management, application security, configurationmanagement, financial metrics

https://benchmarks.cisecurity.org/downloads/browse/?category=metrics





Top 20 Critical Security Controls

V5.0 published February 2014Identifies controls having greatest positive impact on riskposture

Includes suggested metrics for most controls

http://www.councilonc ybersecurity.org

http://www.counciloncybersecurity.org/







Also, check out these conference sessions…



What are the challenges?

Lack of common vocabulary and definitions

We don’t speak the language of executives:Institutional goalsRisksROI

Finally, practical guidance and examples!




Improvements

Where do Istart?




Seven-Step Methodology

Review and refine

Create action plan

Determine how to report

Establish benchmarks and targets

Develop strategies for generation

Decide what metrics to generate

Define goal(s) and objectives



Step 1Define the metricsprogram goal(s) andobjectives

Clearly state the end toward whichall metrics and measurements shouldbe directed

Indicate high level actions that mustbe collectively accomplished to meetthe goal(s)



Step 2Decide what metrics togenerate

Use existing process improvementframework to determine metrics



Framework Examples

Six Sigma Breakthrough StrategyBalanced ScorecardEnterprise Risk Management

Enterprise-level Compliance TrackingStrong Focus Within Institution On:

ROIOn time/on schedule project completion

National rankingsBond ratingsEtc.



Step 2Decide what metricsto generate

Use existing process improvementframework to determine metrics

In the absence of pre-existing

framework, use top-down orbottom-up approach for determiningwhat metrics might be desirable



Top-down Approach

S TEPS E XAMPLES

a. Define/list objectives of the overallsecurity program

To reduce the number of virus infectionswithin the institution by 30% by 2015

b. Identify metrics that would indicate progress toward each objective

Current ratio of virus alerts to actualinfections as compared to the baseline2012 figure

c. Determine measurements needed foreach metric

Number of virus alerts issued to theorganization by month

Number of virus infections reported



Bottom-up Approach

S TEPS E XAMPLES

a. Identify measurements that are/could be collected for this process

Average number of criticalvulnerabilities detected monthly in

servers using xyz scanning tool

b. Determine metrics that could begenerated from the measurements

Change in number of criticalvulnerabilities detected in servers since

xyz scanning tool implemented

c. Determine the association between thederived metrics and establishedobjectives of the overall security program

To reduce the number of detectablevulnerabilities on servers by 95% by2015.



Step 3Develop strategies forgenerating the metrics

Identify trustworthy sources of dataInternal, e.g., IT operations, Audit, Risk Management,Finance, Compliance, etc.External, e.g., actuarial data, annual breach stats, etc.

Decide on frequency of data collectionAssign responsibility for assuringaccuracy of raw data

Develop methods for compiling data intomeasurements and generating metrics





Step 5Determine how themetrics will bereported

Effective communication of metrics isobviously key. Don’t over -simplify, butpresent clearly.

Vary what is reported and howdepending upon audience

Determine context, format,frequency, distribution method, and

reporting responsibility



Step 6Create an action planand act on it

Plan and conduct actions needed togenerate metrics; test, verify,investigate anomalies; implementDocument!

Field Data



Measure ID ISPA-1

Goal Identify levels of serious network-based threats of the kind monitored by the FireEye scanner bynetwork

Measure Since implementation of FireEye monitoring, number of high severity level infections detected on eachnetwork

Type Implementation

Formula Number of critical issues identified by FireEye on each network

Target Baseline; comparison

Definition of Measures

Contributing to Metrics

•“Critical” is defined by FireEye based on its risk analysis; it represents items receiving a score of 5-7 on

a 7 point scale; UVa does not have input on this definition•The networks are defined by the UVa network architecture; the number of devices on each networkwill vary over time as devices are added and migrated between networks; additional networks may beadded to the list of those scanned over time

Frequency Data are collected daily; they will be reported, as appropriate, on a daily, weekly, monthly, quarterlyand/or fiscal year basis

Responsible Parties •Information Owner: AVP ISPRO•Information Collector: ISPA team•Information Customer: VP/CIO, AVP ISPRO, Director ISPA

Data Source FireEye console

Reporting Format Bar graph; spreadsheet



Step 7Establish a formalprogram review andrefinement cycle

Doubt about metric accuracy?

Value worth effort to generate?

New metric best practices/guidanceto consider?

Most important: did metrics guideimprovements to overall securityprogram?



Adjust for Maturity of Security Program



Usefulness of a Given Metric Varies Depending Upon



Maturity of the Security Program

PoliciesDeveloped

ProceduresDeveloped

Procedures &Controls

Implemented

Procedures &ControlsTested

Procedures& ControlsIntegrated

Useful metrics difficult to produce at this early

stage; limited availability of data and collectionmay be difficult





PoliciesDeveloped

ProceduresDeveloped


Implemented



Primary focus on implementationmetrics• Ex: Increase in # of departments that have

mission continuity plans





PoliciesDeveloped

ProceduresDeveloped


Implemented



Primary focus on efficiency andeffectiveness metrics• Ex: % of total departments with updated, tested

mission continuity plans





PoliciesDeveloped

ProceduresDeveloped


Implemented



Primary focus on impact metrics

• Ex: Outcome of 48-hour power outage inadministration bldg.





PoliciesDeveloped

ProceduresDeveloped


Implemented



Primary focus on implementationmetrics• Ex: Sensitive data scanning tool deployed on all

individual desktops/laptops





PoliciesDeveloped

ProceduresDeveloped


Implemented



Primary focus on efficiency and

effectiveness metrics• Ex: # of unapproved storage of sensitive data found

on desktops/laptops





PoliciesDeveloped

ProceduresDeveloped


Implemented



Primary focus on impact metrics

• Ex: Reduction in sensitive data exposures dueto stolen or vulnerable desktops/laptops



Where do I start?

Leverage existing frameworks for expressing: progresstoward goals, value propositions, process improvements,etc.

Use systematic approach for defining effective metricsAdjust metric types as security program matures

ImplementationMetrics

Efficiency andEffectivenessMetrics

Impact Metrics



DemonstrateValue & DriveImprovements

How should Icommunicate

metrics?


G d N It’ N A H tT i



Good News: It’s Now A Hot Topic





84

But, how to make your message heard?



Albert Einstein

Things shouldbe made assimple aspossible, but notany simpler



Tip…

Customize yourmetrics-basedinformation forthe audience

Customize for Security Engineers



Customize for Security Engineers

• Change in #malwareinfections overtime

• # web app

vulnerabilitiesdetected sincescan toolimplemented

• Mean timebetween phishreport andblocked malicioussites

Customize for CISO/CIO



Customize for CISO/CIO

• Change in # ofproactive securityconsultationscompared to FY13baseline.

• Sinceimplementing webapplicationsecurity scanningservice, # highseverity levelvulnerabilitiesdetected declined90%.

Customize for Executives



Customize for Executives

• % of IT budgetspent on securitycompared to peerinstitutions

• Since institution-

wide SSNremediationproject initiated,change in ratio ofdata securitybreaches to totalsecurity incidentsinvestigated



Tip…

Use effectivevisuals







AdditionalTips for Communication



Additional Tips for Communication

Provide right metrics for issue at hand

Provide brief interpretation and analysis

Use specific audience’s language

Link to business goals and objectives



How should I communicate metrics?

Take heart. You now have a receptive audience.

Tailor for the audience

Delivery method is as important as what you have to say

Right metric clearly conveyed=

Right conclusion & decision



References



References

Allen, Julia. “Making the Business Case for Information Security: Selling to Senior Managements.” Carnegie

Mellon University at InfoSec World, 2003, March 10, 2003

Allen, Julia and Stephani Losi. “The ROI of Security.” Software Engineering Institute, Carnegie MellonUniversity, October 1, 2006. http://resources.sei.cmu.edu/asset_files/podcast/2006_016_100_47182.pdf

Berinato , Scott, “Calculated Risk: Return on Security Investment,” CSOonline.com, December 9, 2002,http://www.csoonline.com/article/2113094/metrics-budgets/calculated-risk--return-on-security-investment.html

Center for Internet Security. “CIS Consensus Information Security Metrics,” November 2010.https://benchmarks.cisecurity.org/downloads/browse/?category=metrics

Council on Cybersecurity . “Top 20 Information Security Controls,” http://www.counciloncybersecurity.org

Cullinane, Dave. “Security Awareness and Communication in the C -Suite,” EDUCAUSE e -Live Webinar,October 4, 2012. http://www.educause.edu/library/resources/security-awareness-and-communication-c-

suiteEDUCAUSE: 7 Things You Should Know About Security Metrics,http://www.educause.edu/library/resources/7-things-you-should-know-about-information-security-metrics

References



References

EDUCAUSE: Guide To Effective Security Metrics,

https://wiki.internet2.edu/confluence/display/2014infosecurityguide/Effective+Security+Metrics

EDUCAUSE: Security Metrics Resource Library, http://www.educause.edu/library/security-metrics

EDUCAUSE Core Data Service, http://www.educause.edu/research-and-publications/research/core-data-service

Hinson , Dr. Gary, “Seven Myths About Security Metrics,” ISSA Journal, July 2006.

http://www.noticebored.com/html/metrics.html

ISO/IEC 27004 http://www.iso.org/iso/catalogue_detail?csnumber=42106

Jansen, Wayne. “Directions in Security Metrics Research,” NISTIR 7564; April 2009.http://csrc.nist.gov/publications/nistir/ir7564/nistir-7564_metrics-research.pdf

Jelen, George. “SSE -CMM Security Metrics.” NIST and CSSPAB Workshop, Washington, D.C., 13 -14 June

2000. URL: http://csrc.nist.gov/csspab/june13-15/jelen.pdf (10 July 2001).

Payne , Shirley C., “A Guide To Security Metrics,” SANS Reading Room, July 11, 2001, updated June 19,2006. http://www.sans.org/reading-room/whitepapers/auditing/guide-security-metrics-55

References



References

“Performance Measurement Guide for Information Security,” NIST SP 800 -55 Revision 1 – http://csrc.nist.gov/publications/PubsSPs.html

Poneman Institute. “The 2013 Cost of a Data Breach: Global Analysis,” May 28, 2013.http://www.ponemon.org/library/2013-cost-of-data-breach-global-analysis

Ponemon Institute. “The State of Risk -based Security Management 2013,”http://www.tripwire.com/ponemon/2013/

Schneier, Bruce. “Security ROI,” September 2, 2008https://www.schneier.com/blog/archives/2008/09/security_roi_1.html

Slater, Derek, “Security Metrics: Critical Issues,” CSOonline.com, Nov 12, 2012,http://www.csoonline.com/article/2123361/metrics-budgets/security-metrics--critical-issues.html

Stafford, Eugene and Christina Torode. “A bleak picture of IT security metrics and fighting maliciousattacks,” ISSA conference, Nashville, Tenn., December 11, 2013http://searchcompliance.techtarget.com/video/A-bleak-picture-of-IT-security-metrics-and-fighting-malicious-attacks

Most Of All…



Most Of All…

Keep your eyes on the forest, not the trees!

Documents

SEM01A-Using Information Security Metrics to Demonstrate Value and Drive Improvements (separate registration required) (236663454)