MOVING MOUNTAINS THROUGH MEASUREMENT

Chris ClymerDirector of Security

MRK

Jack NichelsonDirector of Infrastructure & Security

Chart Industries

Jason MiddaughDirector of Infrastructure& Security

Cliffs Natural Resources

INTRODUCTION

Why are we here? What are our goals? What will you gain from this presentation?

WHAT WILL YOU GET FROM THIS PRESENTATION?

A repeatable process for measuring security performance A playbook you can start executing on tomorrow Methods for explaining security to your CEO Tools for justifying security budget

WHAT WONT YOU GET FROM THIS PRESENTATION?

Our favorite metrics Details on how to build the best spreadsheet Instructions on using fancy reporting tools

CHRIS CLYMER Director of Security Services for MRK

CISO for companies ranging from SMB’s to multi-billion dollar corporations

Former board member for NEOISF & co-host of the Security Justice podcast

Aspiring Ironman, amateur saberist

JASON MIDDAUGH

Director of Infrastructure & Security Services at Cliffs Natural Resources

Holds a bachelor’s degree in information systems/operations management from the University of Toledo, an executive master’s degree in business administration from The Ohio State University, and holds several IT based certifications (CISSP, MCSE, VCP, CCNA, CCDA).

Chair member of the North East Ohio Cyber Consortium. Selected and currently attending the FBI Citizens Academy.

Skiing enthusiast

Jack Nichelson “Solving Problems, is my Passion”

Director of Infrastructure & Security for Chart Industries.

Executive MBA from Baldwin-Wallace University

Recognized as one of the “People Who Made a Difference in Security” by the SANS Institute and Received the CSO50 award for connecting security initiatives to business value.

Adviser for Baldwin Wallace’s, State winner Collegiate Cyber Defense Competition (CCDC) team.

I defend my companies competitive advantage by helping solve business problems through technology to work faster and safer.

JACK NICHELSON

WHY METRICS?

WHY METRICS?

Can only improve what you can measure Without measurement, security is purely artistry and witchcraft Because security wants a seat at the big table

Business units that report to the board will have some way to demonstrate their performance Because you need to know what’s working, and what needs your

attention Because you want to buy more security toys next year Because you want to add more staff

HOW I’VE USED THEM

Drive team performance Explain security to the board Justify past expense Justify FUTURE expense Keep myself honest

HOW I’VE USED THEM

Dec-14 Jan-15 Feb-15 Mar-15 Apr-15 May-15 June-15 July-15 Aug-15 Sept-15 Oct-15 Nov-150

50000

100000

150000

200000

250000

4,600 8,190 6,200 6,900 9,100 9,100 4,600 4,800 7,600 8,425 7,53216,968 15,786 15,865 26,661 30,449

28413,893

2,018 2,009 2,675 3,610 4,00962 103

638 1,727

3,030

35,864 62,404

105,617 112,804125,653 132,645

11,009 15,900 17,00616,101

20,782

14,680

12,910

63,501

55,69610,850

84,63090,752

14 64 3412

17

25

46

32

11

11

85

32,591 40,002 39,20850,312

62,075

18,019

71,813

132,555

168,133

133,940

222,326234,943

Total # of New Security Incidents

Total # of Malicious Websites Auto-Blocked

Attacks Auto-Prevented by IPS(Critical/High)

Total # of Malware Auto-Handled

Total # of Phishing Auto-Handled

• IPS Tuned

• AV Signature Issue

• New Web Filter

HOW I’VE USED THEM

Dec-14 Jan-15 Feb-15 Mar-15 Apr-15 May-15 June-15 July-15 Aug-15 Sept-15 Oct-15 Nov-150

50000

100000

150000

200000

250000

4,600 8,190 6,200 6,900 9,100 9,100 4,600 4,800 7,600 8,425 7,53216,968 15,786 15,865 26,661 30,449

28413,893

2,018 2,009 2,675 3,610 4,00962 103

638 1,727

3,030

35,864 62,404

105,617 112,804125,653 132,645

11,009 15,900 17,00616,101

20,782

14,680

12,910

63,501

55,69610,850

84,63090,752

14 64 3412

17

25

46

32

11

11

85

32,591 40,002 39,20850,312

62,075

18,019

71,813

132,555

168,133

133,940

222,326234,943

Total # of New Security Incidents

Total # of Malicious Websites Auto-Blocked

Attacks Auto-Prevented by IPS(Critical/High)

Total # of Malware Auto-Handled

Total # of Phishing Auto-Handled

• IPS Tuned

• AV Signature Issue

• New Web Filter

WHAT MAKES A GOOD METRIC?

Easily measured Easily understood Enables decision-making Meaningful Consistent Quantitative Aligns with your

organization’s Risk Profile

MY PROCESS

1. Define the problem

2. Identify your resources

3. Build a rough draft

4. Review with stakeholders

5. Rinse & repeat

MY PROCESS

1. Define the problema. Where is the pain?b. Find your stakeholdersc. What is working/what isn’t?

2. Identify your resourcesa. What is readily available?b. Tools, process, peoplec. Don’t dwell on nice-to-haves

3. Build a rough drafta. Won’t be perfect the first timeb. Make sure its easy to repeat

4. Review with stakeholdersa. Above and belowb. Have them contributec. Get them to buy-in

5. Rinse & repeata. Run the processb. See what works and what doesn’tc. Adjust where needed

JASON'S STORY

FIRST STEPS – GETTING STARTED

Know your audience and speak their language

Leverage existing tools that you already have

Anticipate what others will ask or what conclusions they will draw from the metrics you are presenting

PAST PROJECTS AND SPEND

IT Executives and Board members want to see that they getting results from previous initiatives

Use these slides to create a “Wow” factor

CURRENTLY MEASURED METRICS

Your goal should be to eventually get to at least 13 months of revolving data

Play the Game of Thrones, or in this case the Game of Met-tricks

LEVERAGE METRICS FOR YOUR FUTURE NEEDS

Use your metrics to help justify additional capital

Help predict when you are going to run out or need more

Show capability gaps that you want to fill

Justify additional resources

HOW AND WHERE DO I GET STARTED?

Kick things off in your next team meeting

Don’t worry about the past

Don’t get hung up on 100% accuracy

You don’t have to hit the bulls-eye on every metric, shoot for the inner-rings

SUMMARY

Know your audience - Speak their language

Think Past – Leverage metrics to justify and put historic spend at ease

Think Present – Collect and analyze your data; do not be afraid to burn it down and start again

Think Future – Continually set the stage and constantly communicate when and where you are going to need additional resources with metrics

Get Started! – You do not have to hit the bullseye

JACK'S JOURNEY

BE PROACTIVE

Change starts from within, so you have to make the decision to focus on the things you can influence rather than reacting to the things outside of your control.

Manage Yourself: Where and how are you spending your time & energy throughout the day? Make a list of the things that concern you and things you can Influence.

Ask yourself these 3 questions every day: Did I do my best to spend my time on things I can influence? Did I do my best to set and communicate clear goals? Did I do my best to make progress toward goal achievement?

“The 1st metric you need to track is yourself”

BEGIN WITH THE END IN MIND

If your ladder is not leaning against the right wall, every step you take gets you to the wrong place faster.First, do you know what “good” looks like? Break down the area you have influence over into functional parts

that you and the stockholders can score and rank.

Now that you have an agreed upon heatmap of your current state, set short term and long term goals.

“Try Not to Become a Success. Rather Become a Person of Value.”

PROBLEM STATEMENT

The Problem Statement significantly clarifies the current situation by specifically identifying the problem and its severity, likelihood, and impact. It also serves as a great communication tool, helping to get buy-in and support from others.Build & Execute plans to drive for results & share successes Invest more time in project planning and due diligence; time spent defining

the problem is NEVER time wasted.

Write a Project Charter, clearly state the scope, objectives, participants, and success measurements.

Create a Work Breakdown Structure to graphically represent the project scope, broken down in successive chunks with defined deliverables.

“People with a plan succeed because they know where they’re going”

PUT FIRST THINGS FIRST

Focus on the important, not just the urgent. The urgent are not that important, and the important are never urgent.

Tips for taking back control of your time: Stop saying Yes, When you want to say No.

Scheduled your own time with purpose & defend it!

Don’t be afraid to close your email and turn off your phone

“Effectiveness requires the integrity to act on your priorities”

CHART PERFORMANCE & ADJUSTGemba ( 現場 ) is a Japanese term referring to the place where value is created. The idea of Gemba is that the problems are visible, and the best improvement ideas will come from going to the Gemba.

“Good security is not something you have, it’s something you do” - Wendy Nather

SUMMARY – KEY TAKEAWAYS Be Proactive – Focus on what you can influence

Begin with the end in mind – Define practical outcomes

Create a Problem Statement – A goal without a plan is just a wish

Put first thing first – Plan weekly, act daily

Chart Performance & Adjust – Shine a light on the problem

“Think about how you can simplify security – make it easy – and focus on the basics.” - Dave Kennedy

WRAPPING UP

OUR 3 APPROACHES

Chris Jason Jack

1.Define the problem

2.Identify your resources

3.Build a rough draft

4.Review with stakeholders

5.Rinse & repeat

1.Know your audience

2.Think Past

3.Think Present

4.Think Future

5.Get Started!

1.Be Proactive

2.Begin with the end in mind

3.Create a Problem Statement

4.Put first thing first

5.Chart Performance & Adjust

THANK YOU

Chris ClymerChris@ChrisClymer.com

@ChrisClymer

Jack NichelsonJack@Nichelson.net

@Jack0lope

Jason MiddaughJason.Middaugh@CliffsNR.com

@Jason_Middaugh

APPENDIX

REFERENCES

Security Metrics – Andrew Jacquith Security Data Visualization – Greg Conti Pragmatic Security Metrics – W. Krag Brotby & Gary Hinson Security Metrics Mailing list - http://www.securitymetrics.org/mailing-list.html NACD Cyber Risk Oversight Handbook 2016 - NACD

HOW TO AUTO-LINK EXCEL GRAPH (OR CELLS) TO POWERPOINT DECK

Open Excel Click on Graph (or Cells) and hit Copy In PowerPoint select Paste Paste Special … Change the radio button to “Paste link” and highlight “Microsoft Excel Chart Object” and

press OK The Chart (or Cell) with appear in the PowerPoint Deck and with auto-update from the

source Excel file each time the PowerPoint Deck is opened

Notes: I highly recommend you save both the Excel and PowerPoint files in shared workspace

like SharePoint Do not rename the Excel file, it will break the links Do not move the location of the Excel file, it will break the links Do not rename the tabs in the Excel file, it will break the links If you delete and recreate the graph in Excel, you will need to re-link it to the PowerPoint

presentation (the graph will have a new name)

HOW TO BUILD A SQCD BOARD

Key Performance Indicators – Good data can tell a story Predictive Analysis – Your board should help prevent future issues Keep the data fresh and useful, address items as quick as possible

using LEAN tools and once addressed remove them from the board.

GEMBA BOARD: SECURITY“We measure things that matter”

Example Metrics: # of systems not monitored & tracked in inventory by

Location or LoB # Top Vulnerabilities by Location or LoB # of Legacy Systems by Location or LoB # of Users with Local Admin & Accounts with Domain Admin # of Total Security Incidences by Location or LoB # of Past Due Security Awareness Training by Location or

LoB

Security - The current security posture at a glance

GEMBA BOARD: QUALITY

Example Metrics: # of Servers & Workstation missing OS & App patches

(30 day SLA) # of infections/Re-Images tickets (3 day SLA) # of Security Event tickets (5 day SLA) # of Security Request tickets (15 days SAL) Cause Mapping Analysis to find root cause of problems

Quality – Results for SLA goals of events & requests

GEMBA BOARD: DELIVERYDelivery – Active Projects & Audits at a glanceExample Metrics: Active Projects Status Active Audit Status Remediation Progress by Location or LoB On-Site Awareness Training by Location

GEMBA BOARD: COST

Cost – P&L at a glanceExample Metrics: Operating budget spending plan (OPEX & CAPEX) ROIC Qualitatively Rating of Perceived Value Support Agreements Costs & Renew dates Consultant Support Agreements Costs & Renew dates Running total of cost savings

GEMBA BOARD: PEOPLE

People – Skills matrix at a glanceExample Metrics: Skills Matrix of everyone in Security Training and development plans On-Call & Vacation Schedules Awards

VISUALIZATION TECHNIQUES: THE HEATMAP

ImpactLow No threat to core business function impact

MediumThreat to core business function impact, but has

not occurred yet. i.e. ERP system is down but have not yet missed orders

High Immediate impact to core business functions. i.e. products cannot be shipped, or core IP is lost.

Likelihood

Low Happens once every 10 years, or less

Medium Happens once every 1 to 10 years

High Happens once or more a year

• Develop “Likelihood” to fit your org• Develop “Impact” to fit your org”• Score potential risks “high”,

“medium”, or “low” for each• Map results to the heatmap

VISUALIZATION TECHNIQUES: RISK REGISTER

VISUALIZATION: MULTI-LAYERED DEFENSE STRATEGY

95% of systems are patched within 30 days.

Once discovered - improve tools.

Block about 27,500 intrusions & 880 virus / month

Detect & fix 20 viruses & 3,500 exploits / month

Block 1.6 million emails & Stop 22,000 virus / month

Successful

Unknown Attacks

IPSSpam Filter

Admin Rights RemovalAnti Virus & MalwareBytes

Employee User Security TrainingHardware refresh & Laptop Encryption

Password managementPatching strategy

Outside independent measurement of security outcomes

Block 1.6m /month bad IP’sand 600,000 / month bad URLs

Contain

Prevent

Independent penetration testingAttackers

BitSight Score

Firewall / Open DNS IP Filter / Web Filter

2600 Users no longer have Admin Rights

97% of Laptops are now encrypted

VISUALIZATION TECHNIQUES: CLUSTERED STACKED GRAPH

Dec-14

Jan-15

Feb-15

Mar-15

Apr-15

May-15

June-15

July-15

Aug-15

Sept-15

Oct-14

Nov-14

0

5

10

15

20

3 4 3 3 4 5 4 4 4 4

6 33 2

2

44 3 4

2

68

88

8

8

7

66

2

2 22

22

2

2

22

2

17 1716

1516

19

17

1516

10 # of Open Low Vulns# of Open Moderate Vulns# of Open Important Vulns# of Open Critical Vulns

Good for showing portions of a whole over time

Quick visual indicator on where things stand

Examples: Vulnerabilities by

criticality/month Incidents by criticality/month Total block events (AV, FW,

Proxy, etc.)

VISUALIZATION TECHNIQUES: CLUSTERED STACKED GRAPH

Steps to create: Select your data, and create a

“stacked column chart” in Excel Make sure each data element is

setup as a “stacked column” To create a total for each column,

include this in your source data, and enter it as a “Line” under chart type. Give the line a blank “fill” and “line” and you’ll get the numbers only

Be sure your data includes a total row

For more, see: http://www.exceldashboardtemplates.com/how-to-easily-create-a-stacked-clustered-column-chart-in-excel/

VISUALIZATION TECHNIQUES: THE SCORECARD

Captures day-to-day operations in security One-page roll-up that can be presented to CIO, or used internally “Operations” section captures work being done: creating firewall rules,

patching systems, conducting awareness events The “Risk” section captures visibility into risk at the organization.

Number of scans, open vulnerabilities To the far right is the legend explaining the thresholds for each item

VISUALIZATION TECHNIQUES: THE SCORECARD