SESSION ID:

Succeeding with Enterprise Software Security Key Performance Indicators

ASEC-T08

Rafal M. Los Principal, Strategic Security Services

HP Enterprise Services @Wh1t3Rabbit

Introduction to Key Performance Indicators (KPIs)

#RSAC

Reporting on progress is tricky

3

#RSAC

If you spend $1M, then…?

4

#RSAC

First things first… Who here reports metrics?

5

#RSAC

How many metrics do you track?

6

#RSAC

I was once a victim of metrics

7

#RSAC

Do your metrics give you insight?

8

#RSAC

KPIs do.

9

#RSAC

KPI = Key Performance Indicator

#RSAC

A key performance indicator (KPI) is a measure of performance, commonly used to help an organization define and evaluate how successful it is, typically in terms of making progress towards its long-term organizational goals.

11

#RSAC

…but implies you have long-term organizational goals!

12

#RSAC

TL;DR: “Are you succeeding?”

13

#RSAC

..and how much, relative to goals?

14

#RSAC

Trademarks of good KPIs:

15

#RSAC

1) Show relative distance to a goal

16

#RSAC

2) Establish relevance to org

17

#RSAC

3) Establish relevance to security

18

#RSAC

>> context <<

19

metrics vs KPIs

#RSAC

How do you convey “improving”?

#RSAC

Improvement as a result of effort

22

#RSAC

Easy right? So why are we so bad at it?

23

#RSAC

More importantly…

24

#RSAC

..how do you define success?

25

#RSAC

Study the following graph:

26

#RSAC

27

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

Q1 2012 Q2 2012 Q3 2012 Q4 2012 Q1 2013 Q2 2013 Q3 2013 Q4 2013

Issues by OWASP Top 10

A5 A4 A3 A2 A1

#RSAC

What does it show?

28

#RSAC

Look again…

29

#RSAC

30

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

Q1 2012 Q2 2012 Q3 2012 Q4 2012 Q1 2013 Q2 2013 Q3 2013 Q4 2013

Issues by OWASP Top 10

A5 A4 A3 A2 A1

A B D C

#RSAC

A: Implemented mandatory testing

31

#RSAC

B: Major acquisition

32

#RSAC

C: Integration into primary dev cycle

33

#RSAC

D: Switched s/w sec testing tools

34

#RSAC

Clearly, the graph is inadequate

35

#RSAC

Raw data:

36

Q1 2012 3575 135 4387 135 237

Q2 2012 3250 87 4357 31 219

Q3 2012 2978 12 3648 12 35

Q4 2012 4208 141 7989 47 187

Q1 2013 4189 109 6897 41 24

Q2 2013 2138 71 5867 39 23

Q3 2013 1378 14 2807 31 28

Q4 2013 2366 51 3879 38 31

A1 A2 A3 A4 A5

#RSAC

Q1-Q2 2013: 49% decrease in A1 Q3-Q4 2013: 72% increase in A1

37

#RSAC

Clearly this is data without context

38

#RSAC

This shows no impact

39

Defining effective KPIs

#RSAC

What makes a good KPI?

41

#RSAC

1. Relative distance to goal 2. Relevance to organization 3. Relevance to security

#RSAC

Focus on 4 key SwSec areas

43

#RSAC

“Impact to effort”

44

#RSAC

Impact of a security item to the overall effort of the project

45

#RSAC

[security item] [dev effort]

46

#RSAC

Security items (examples) • static analysis process • dynamic analysis process • integrating testing tools • developer awareness

47

#RSAC

Development effort • person-hours required to

complete existing task

48

#RSAC

“By adding a dynamic testing process we initially added 25% effort but over 4 quarters now only add 10%” – AppSec Prog Mgr

49

#RSAC

50

0%

5%

10%

15%

20%

25%

30%

35%

Q1 2012 Q2 2012 Q3 2012 Q4 2012 Q1 2013 Q2 2013 Q3 2013 Q4 2013

I2E (additional person-hours)

#RSAC

We’re showing that we’re impacting the AppDev process less over time

51

#RSAC

Doesn’t tell us if it’s helping security …

52

?

#RSAC

“Impact to release”

53

#RSAC

Impact of a security item to the release timeline

54

#RSAC

[security item] [release timeline]

55

#RSAC

Security items (examples) • integrating security testing early

in development • providing templates for ‘fixes’ • defining pre-built code modules

56

#RSAC

Release timeline • person-hours required to

complete existing task

57

#RSAC

“We were able to show that we could release faster if security was involved earlier on in development” – AppSec Prog Mgr

58

#RSAC

59

0

10

20

30

40

50

60

70

80

90

100

Q1 2012 Q2 2012 Q3 2012 Q4 2012 Q1 2013 Q2 2013 Q3 2013 Q4 2013

I2R (hours additional avg/project)

#RSAC

We’re showing that we’re impacting the release process less over time

60

#RSAC

“Impact to uptime”

61

#RSAC

Impact of a security item to the uptime of the application/service

62

#RSAC

[security item] [uptime]

63

#RSAC

Security items (examples) • continuous security monitoring • continuous/regular testing • remediation of exploitable vulns

64

#RSAC

Uptime • an application/service event that

causes downtime due to security-related issue (configuration, attack, etc.)

65

#RSAC

“We were able to prove that remediating all discovered SQL injection issues caused less application downtime” – AppSec Prog Mgr

66

#RSAC

67

0

10

20

30

40

50

60

Q1 2012 Q2 2012 Q3 2012 Q4 2012 Q1 2013 Q2 2013 Q3 2013 Q4 2013

I2U (hours of effected uptime total)

#RSAC

We’re showing that removing injection vulnerabilities, which are easily exploitable, reduces downtime.

68

#RSAC

“Impact to residual risk”

69

#RSAC

Impact of a security item to residual risk of an application or service

70

#RSAC

[security item] [residual risk]

71

#RSAC

Security items (examples) • mandatory peer review of code • required stage-gates to

production w/security sign*-off • accountability by LoB VP

72

#RSAC

Residual risk • a level of residual risk in the

application as a result of security effort(s)

73

#RSAC

“For each line of business that reported risk metrics up to the VP successfully, residual risk decreased.” – AppSec Prog Mgr

74

#RSAC

75

0 20 40 60 80 100 120 140 160 180 200

App 1

App 1'

App 2

App 2'

App 3

App 3'

Residual Risk Charting

Cycle 4 Cycle 3 Cycle 2 Cycle 1

App x = Application w/o VP accountability App x’ = Application with VP accountability

#RSAC

We’re showing that raising accountability to the LoB VP, residual risks fall greatly

76

Defining a set of KPIs

#RSAC

What is the goal of your effort?

#RSAC

Minimize injection (A1) defects in new software releases

79

#RSAC

“Let’s show progress”

#RSAC

What security did: Introduced (self-service) static analysis tools into development cycle

81

#RSAC

Impact it had: Initially the impact was prohibitive, but with effort became manageable.

82

#RSAC

“Impact to effort”

83

#RSAC

84

0

10

20

30

40

50

60

70

80

90

100

Q4 2012 Q1 2013 Q2 2013 Q3 2013 Q4 2013

Real I2E (hrs per dev per project)

Dev 1 Dev 2 Dev 3 Dev 4

Q4 2012 Baseline

Q1 2013 Initial rollout

Q2 2013 Product training

Q3 2013 IDE Automation

Q4 2013 Workstream integration

#RSAC

“Impact to release”

85

#RSAC

86

0

20

40

60

80

100

120

140

160

App 1 App 2 App 3 App 4

Real I2E (hrs per dev per project)

Q4 2012 Q1 2013 Q2 2013 Q3 2013 Q4 2013

Q4 2012 Baseline

Q1 2013 Initial rollout

Q2 2013 Product training

Q3 2013 IDE Automation

Q4 2013 Workstream integration

#RSAC

“Impact to uptime”

87

#RSAC

88

0

2

4

6

8

10

12

14

16

18

20

App 1 App 2 App 3 App 4

Security related downtime events

Q4 2012 Q1 2013 Q2 2013 Q3 2013 Q4 2013

Q4 2012 Baseline

Q1 2013 Initial rollout

Q2 2013 Product training

Q3 2013 IDE Automation

Q4 2013 Workstream integration

#RSAC

“Impact to residual risk”

89

#RSAC

90

0

50

100

150

200

250

Q4 2012 Q1 2013 Q2 2013 Q3 2013 Q4 2013

Impact to residual risk *Only A1 + A2 (OWASP Top 10)

App 1 App 2 App 3 App 4

Q4 2012 Baseline

Q1 2013 Initial rollout

Q2 2013 Product training

Q3 2013 IDE Automation

Q4 2013 Workstream integration

*based on organization’s basic IT risk’ calculation

#RSAC

For the adventurous: “Impact to business”

91

#RSAC

Is this approach perfect? No.

92

#RSAC

Do these KPIs work everywhere? No.

93

#RSAC

Better than existing metrics? Absolutely.

94

#RSAC

Strive to do better.

95

Demonstrate meaningful

progress

#RSAC

Follow the wh1t3rabbit.

96

https://twitter.com/Wh1t3Rabbit