Security Predictions

  • View
    380

  • Download
    1

Embed Size (px)

DESCRIPTION

This is a presentation I gave at a workshop that was co-located with ESSoS 2010. The presentation is about using Metrics Validation Criteria to choose a valid predictive metric for security vulnerabilities.

Text of Security Predictions

  • 1. Metrics validation criteria: How do we know when a metric is worthwhile? Ben Smith Andy Meneely Laurie Williams

2. Scenario

  • You and your team are asked to choose a set of metrics for your development companys front-running application, iAwesome.The goal of this metrics project is to reducepost-release vulnerabilitiesby predicting them during the software lifecycle.How do you demonstrate to management that your metrics aremeaningfulandworthwhile ?

3. Metric Uses Metrics Quality Assessment Process Certification Process Improvement Task Planning Research Prediction 4. Motivation Software System Componentm=.25 Componentm=.95 Componentm=. 05 Componentm=.21 Componentm=.15 Componentm=.01 Prediction M < .2 5. Well, the metric was predictive

  • but may not bevalid !
  • How do we know when a metric is valid?

6. Metrics Validation Criteria

  • Metrics validation criteria : boolean statements about various aspects of the validity of a metric.
  • Example:
  • Underlying theory validity : Is there an underlying theory as to why the metric was chosen?

7. Agenda

  • Motivation: what is validity?
  • Anatomy of a systematic literature review
  • Validating a security metric for prediction
  • Is prediction the only answer?

8. Objective

  • Guide researchersin making
  • Sound contributionsto the metrics field
  • Providing apractical summary
  • The superset of all proposed metrics validation criteria

9. Foundation in the Literature 10. Systematic literature review Phase Size of Source List Literature Index 2,228 Title 536 Cross-confirmed Title 156 Abstract 44 Full-text 17 Follow-up 20 11. Results of the Review

  • Three major categories for metrics validation criteria:
    • Internal : the metric correctly measures the attribute it purports to measure.
    • External : the metric is related in some way with an external quality factor.
    • Construct : the gathering of a metrics measurements is suitable for the definition of the targeted attribute.

12. Two Competing Philosophies

  • Goal-driven : philosophy holds that the primary purpose of a metric is to apply it to a software process.
  • Theory-driven : views that the primary purpose of a metric is to gain understanding of the nature of software.

13. Agenda

  • Motivation: what is validity?
  • Anatomy of a systematic literature review
  • Validating a security metric for prediction
  • Is prediction the only answer?

14. Scenario

  • You and your team are asked to choose a set of metrics for your development companys front-running application, iAwesome.The goal of this metrics project is to reducepost-release vulnerabilitiesby predicting them during the software lifecycle.How do you demonstrate to management that your metrics aremeaningfulandworthwhile ?

15. Choosing the best criteria

  • To succeed with this metrics project, you should chose validation criteria that:
    • Help with the accuracy of prediction
    • Prioritize business over knowledge for the sake of knowledge
    • Are absolutely necessary

16. Metrics Validation Criteria A priori validity Actionability Appropriate Continuity Appropriate Granularity Association Attribute validity Causal model validity Causal relationship validity Content validity Construct validity Constructiveness Definition validityDiscriminative powerDimensional consistencyEconomic productivityEmpirical validity External validityFactor independenceImprovement validityInstrument validityIncreasing growth validityInteraction sensitivityInternal consistencyInternal validityMonotonicityMetric ReliabilityNon-collinearityNon-exploitabilityNon-uniformityNotation validityPermutation validityPredictabilityPrediction system validityProcess or Product RelevanceProtocol validityRank ConsistencyRenaming insensitivityRepeatabilityRepresentation conditionScale validityStabilityTheoretical validityTrackabilityTransformation invarianceUnderlying theory validityUnit validityUsability 17. Reduced Metrics Validation Criteria 18. Rejected (and why)

  • A metric hasimprovement validityif the metric is an improvement over existing metrics .
  • A metric hasincreasing growth validityif the metric increases when concatenating two entities together .

19. Accepted (and why)

  • A metric hasusabilityif it can be cost-effectively implement in a quality assurance program .
  • A metric hasinstrument validityif the underlying measurement is valid and properly calibrated .

20. Agenda

  • Motivation: what is validity?
  • Anatomy of a systematic literature review
  • Validating a security metric for prediction
  • Is prediction the only answer?

21. Measurement Theory

  • Metrics can be used as the route to understanding the nature of software and the software development process
  • Rather than a list of components, wed like a list of action items based on a set of theories:applied science
  • Reactive vs. Proactive

22. Questions?