22
1 Metrics validation criteria: How do we know when a metric is worthwhile? Ben Smith Andy Meneely Laurie Williams

Security Predictions

Embed Size (px)

DESCRIPTION

This is a presentation I gave at a workshop that was co-located with ESSoS 2010. The presentation is about using Metrics Validation Criteria to choose a valid predictive metric for security vulnerabilities.

Citation preview

Page 1: Security Predictions

1

Metrics validation criteria: How do we know when a metric is worthwhile?

Ben Smith

Andy Meneely

Laurie Williams

Page 2: Security Predictions

Scenario

You and your team are asked to choose a set of metrics for your development company’s front-running application, iAwesome. The goal of this metrics project is to reduce post-release vulnerabilities by predicting them during the software lifecycle. How do you demonstrate to management that your metrics are meaningful and worthwhile?

2

Page 3: Security Predictions

Metric Uses

3

MetricsMetrics

Quality Assessment

Process CertificationProcess Improvement

Task Planning

Research

Prediction

Page 4: Security Predictions

Motivation

4

Software System

Component Component m=.25 m=.25Component Component m=.25 m=.25

Component Component m=.95 m=.95Component Component m=.95 m=.95

Component Component m=. m=.0505Component Component m=. m=.0505

Component Component m=.21 m=.21Component Component m=.21 m=.21

Component Component m=.15m=.15Component Component m=.15m=.15

Component Component m=.01 m=.01Component Component m=.01 m=.01

Prediction

M < .2

Page 5: Security Predictions

Well, the metric was predictive…

…but may not be valid!

How do we know when a metric is valid?

5

Page 6: Security Predictions

Metrics Validation Criteria

Metrics validation criteria: boolean statements about various aspects of the validity of a metric.

Example:

Underlying theory validity: Is there an underlying theory as to why the metric was chosen?

6

Page 7: Security Predictions

Agenda

• Motivation: what is validity?

• Anatomy of a systematic literature review

• Validating a security metric for prediction

• Is prediction the only answer?

7

Page 8: Security Predictions

Objective

• Guide researchers in making

• Sound contributions to the metrics field

• Providing a practical summary

• The “superset” of all proposed metrics validation criteria

8

Page 9: Security Predictions

Foundation in the Literature

9

Page 10: Security Predictions

Systematic literature review

10

Phase Size of Source List

Literature Index 2,228

Title 536

Cross-confirmed Title 156

Abstract 44

Full-text 17

Follow-up 20

Page 11: Security Predictions

Results of the Review

• Three major categories for metrics validation criteria:– Internal: the metric correctly measures the attribute it

purports to measure.

– External: the metric is related in some way with an external quality factor.

– Construct: the gathering of a metric’s measurements is suitable for the definition of the targeted attribute.

11

Page 12: Security Predictions

Two Competing Philosophies

• Goal-driven: philosophy holds that the primary purpose of a metric is to apply it to a software process.

• Theory-driven: views that the primary purpose of a metric is to gain understanding of the nature of software.

12

Page 13: Security Predictions

Agenda

• Motivation: what is validity?

• Anatomy of a systematic literature review

• Validating a security metric for prediction

• Is prediction the only answer?

13

Page 14: Security Predictions

Scenario

You and your team are asked to choose a set of metrics for your development company’s front-running application, iAwesome. The goal of this metrics project is to reduce post-release vulnerabilities by predicting them during the software lifecycle. How do you demonstrate to management that your metrics are meaningful and worthwhile?

14

Page 15: Security Predictions

Choosing the best criteria

To succeed with this metrics project, you should chose validation criteria that:– Help with the accuracy of prediction– Prioritize business over knowledge for the sake

of knowledge– Are absolutely necessary

15

Page 16: Security Predictions

Metrics Validation Criteria

16

A priori validityActionabilityAppropriate ContinuityAppropriate GranularityAssociationAttribute validityCausal model validityCausal relationship validityContent validityConstruct validityConstructivenessDefinition validity Discriminative power Dimensional consistency Economic productivity Empirical validityExternal validity Factor independence Improvement validity Instrument validity Increasing growth validity Interaction sensitivity Internal consistency Internal validity

Monotonicity Metric Reliability Non-collinearity Non-exploitability Non-uniformity Notation validity Permutation validity Predictability Prediction system validity Process or Product Relevance Protocol validity Rank Consistency Renaming insensitivity Repeatability Representation condition Scale validity Stability Theoretical validity Trackability Transformation invariance Underlying theory validity Unit validity Usability

Page 17: Security Predictions

Reduced Metrics Validation Criteria

17

Page 18: Security Predictions

Rejected (and why)

• A metric has improvement validity if the metric is an improvement over existing metrics.

• A metric has increasing growth validity if the metric increases when concatenating two entities together.

18

Page 19: Security Predictions

Accepted (and why)

• A metric has usability if it can be cost-effectively implement in a quality assurance program.

• A metric has instrument validity if the underlying measurement is valid and properly calibrated.

19

Page 20: Security Predictions

Agenda

• Motivation: what is validity?

• Anatomy of a systematic literature review

• Validating a security metric for prediction

• Is prediction the only answer?

20

Page 21: Security Predictions

Measurement Theory

• Metrics can be used as the route to understanding the nature of software and the software development process

• Rather than a list of components, we’d like a list of action items based on a set of theories: applied science

• Reactive vs. Proactive

21

Page 22: Security Predictions

22

Questions?