Embed Size (px)
DESCRIPTION
Michael Kass National Institute of Standards and Technology http://samate.nist.gov/ [email protected]. Software Assurance Metrics and Tool Evaluation (SAMATE). Outline. Overview of Software Assurance (SwA) tool testing at NIST Description of SAMATE project Follow-on. - PowerPoint PPT Presentation
Citation preview
Software Assurance Metrics and Tool Evaluation (SAMATE)
Michael KassMichael KassNational Institute of National Institute of Standards and TechnologyStandards and Technology
http://samate.nist.gov/http://samate.nist.gov/
[email protected]@nist.gov
June 8, 2005 2
OutlineOutline
Overview of Software Assurance Overview of Software Assurance (SwA) tool testing at NIST(SwA) tool testing at NIST
Description of SAMATE projectDescription of SAMATE project Follow-onFollow-on
June 8, 2005 3
Dept Homeland Security Dept Homeland Security ConcernConcern
Do software assurance tools work as they Do software assurance tools work as they should?should?
Do they really find vulnerabilities and Do they really find vulnerabilities and catch bugs? How much assurance does catch bugs? How much assurance does running the tool provide?running the tool provide?
Software Assurance tools should be:Software Assurance tools should be: Tested (accurate and reliable) Tested (accurate and reliable) Peer reviewedPeer reviewed Generally acceptedGenerally accepted
June 8, 2005 4
Goals of SAMATEGoals of SAMATE
Develop metrics for the effectiveness of Develop metrics for the effectiveness of SwA tools and to identify deficiencies in SwA tools and to identify deficiencies in software assurance methods and tools software assurance methods and tools
Perform SwA R&D to assess current Perform SwA R&D to assess current methods and tools in order to identify methods and tools in order to identify deficiencies which can lead to software deficiencies which can lead to software product failures and vulnerabilitiesproduct failures and vulnerabilities
Identify gaps in methods and tools and Identify gaps in methods and tools and suggest areas of researchsuggest areas of research
June 8, 2005 5
The NIST SAMATE ProjectThe NIST SAMATE Project(Software Assurance Metrics and Tool Evaluation)(Software Assurance Metrics and Tool Evaluation)
1.1. Conduct surveysConduct surveys ToolsTools Researchers and companiesResearchers and companies
2.2. Host workshops & conference sessionsHost workshops & conference sessions Taxonomy of SwA functions and techniquesTaxonomy of SwA functions and techniques Order of importance (cost/benefit, criticalities, …)Order of importance (cost/benefit, criticalities, …) Gaps and research agendasGaps and research agendas Studies to develop tool effectiveness metricsStudies to develop tool effectiveness metrics
3.3. Evaluate toolsEvaluate tools Detailed specificationDetailed specification Test plans Test plans Host reference dataset libraryHost reference dataset library
June 8, 2005 6
A Taxonomy of Static Analysis Tool A Taxonomy of Static Analysis Tool FunctionsFunctions
LanguageLanguage Source/Binary analysisSource/Binary analysis Semantic checking (abstract syntax tree)Semantic checking (abstract syntax tree) Interprocedural analysisInterprocedural analysis Strong type checking (type casting vulnerabilities, uninitialized variable use)Strong type checking (type casting vulnerabilities, uninitialized variable use) Memory allocation checking (memory leaks, deallocation of unallocated memory)Memory allocation checking (memory leaks, deallocation of unallocated memory) Logic checking (unnecessary code, unreachable code)Logic checking (unnecessary code, unreachable code) Interface checking (include file cycling)Interface checking (include file cycling) Security checking Security checking
Buffer overflow/underflowBuffer overflow/underflow Stack overflowsStack overflows Heap overflowsHeap overflows
Integer overflow/underflowInteger overflow/underflow Tainted dataTainted data Error path problemsError path problems Locking problemsLocking problems
Code metric generation (LOC, number of methods, levels of inheritance)Code metric generation (LOC, number of methods, levels of inheritance)
June 8, 2005 7
SA Tool Effectiveness MetricsSA Tool Effectiveness Metrics
What constitutes a tool’s effectiveness metric?What constitutes a tool’s effectiveness metric?
Number of defects detected vs. total defectsNumber of defects detected vs. total defects Number of false positivesNumber of false positives Number of false negativesNumber of false negatives ……
June 8, 2005 8
Documenting tool effectivenessDocumenting tool effectiveness
Tool functional specificationTool functional specification Test planTest plan Reference datasetReference dataset Test reportTest report
June 8, 2005 9
T(mos.) 1 2 6 12 18 24
Workshop1SA
classes
3 4 5 9 15 21
Workshop 3DefineMetric
Workshop 2fill
gaps
focusgroupclass 1
focusgroupclass 1
FunctionTaxonomy
ToolSurvey
SurveyPublication
selectfunc
strawmanspec
test plan
test plandraft
Spec0
Spec1
test plan
test plan
select func
strawmanspec
draft
Spec0
Spec1
SAMATE Project Timeline
focusgroupclass 2
focusgroupclass 2
tool testing matrix
test reports
test reports
test reports
test reports
June 8, 2005 10
Contact for SAMATE Contact for SAMATE ParticipationParticipation
Paul BlackPaul Black Project Leader, Software Diagnostics & Project Leader, Software Diagnostics &
Conformance Testing Division, Software Conformance Testing Division, Software Quality GroupQuality Group
[email protected]@nist.gov
