Upload
others
View
9
Download
0
Embed Size (px)
Citation preview
©2016 Lockheed Martin Aeronautics Company
Tool Output Integration Framework Enhanced Static Code Analysis Identifying Critical Vulnerabilities in Code
Dr. Ben Calloni, P.E., CISSP, CEH Principle Investigator Fellow Software Security OMG’s Cyber Risk Summit September 14, 2016
©2016 Lockheed Martin Aeronautics Company
Overview
• Purpose of the Study − Background
• Previous Lockheed Martin Aeronautics SCA studies • NSA SCA Test Results
− Software Languages Covered • Test Case Coverage
− Common Weakness Enumerations (CWE) by MITRE − NSA Standardized Test Suites (Juliet)
• Tool Capability Analysis Results − 2012 Tool Study – Tool A and Tool C − TOIF Results C/C++
• Summary of Findings • Conclusion
Chart 1
September 14, 2016 OMG Cyber Risk Summit
©2016 Lockheed Martin Aeronautics Company
Purpose
• Not Dynamic Analysis (Testing while executing the code) • Not Penetration Testing
• Incorporate Static Code Analysis within the larger Trusted
Software Development Process − Solid SwE based on
• Requirements • Design • Secure Coding Standards • System Testing and Evaluation
• Make the SCA execution as seamless as the software compile and build “DURING DEVELOPMENT”!
• Eliminate as many security coding flaws at the point of creation.
Chart 2 September 14, 2016 OMG Cyber Risk Summit
©2016 Lockheed Martin Aeronautics Company
Security Definition
Cyber Vulnerability (CISSP BoK) 1. A flaw* (aka weakness) exists in the
system 2. Attacker has access to the flaw, and 3. Attacker has capability to exploit the flaw
• Examples − Lack of security patches − Lack of current virus definitions − Software Bug − Lax physical security
Basic definition of Vulnerability • refers to the inability to withstand the effects of a hostile
environment • open to attack or damage Defenders can only control these!
*e.g. Buffer Overflow is still on SANS Top 25 (#3). Industry has known and discussed since 1988!
September 14, 2016 OMG Cyber Risk Summit Chart 3
©2016 Lockheed Martin Aeronautics Company
Test Case Coverage
• Common Weakness Enumerations (CWE) by MITRE
Chart 4
CWE™ International in scope and free for public use, CWE provides a unified, measurable set of software weaknesses that is enabling more effective discussion, description, selection, and use of software security tools and services that can find these weaknesses in source code and operational systems as well as better understanding and management of software weaknesses related to architecture and design.
• Juliet Test Suite from National Security Agency
The Juliet Test Suite is an aggregation of test cases developed by the National Security Agency (NSA) Center for Assured Software (CAS) specifically for use in testing static analysis tools. It is intended for anyone who wishes to use the test cases for their own testing purposes, or who would like to have a better understanding of how test cases were created. The Juliet Test Suite is comprised of C/C++ and Java test cases. Version 1.1 of the C/C++ test suite contains examples for 119 different CWEs and contains 57,099 test cases. Version 1.1.1 of the Java test suite contains examples of 113 different CWEs and contains 23,957 test cases.
September 14, 2016 OMG Cyber Risk Summit
©2016 Lockheed Martin Aeronautics Company
CWE Example
Chart 5
CWE-119: Improper Restriction of Operations within the Bounds of a Memory Buffer Description Summary The software performs operations on a memory buffer, but it can read from or write to a memory location that is outside of the intended boundary of the buffer. Extended Description Certain languages allow direct addressing of memory locations and do not automatically ensure that these locations are valid for the memory buffer that is being referenced. This can cause read or write operations to be performed on memory locations that may be associated with other variables, data structures, or internal program data. As a result, an attacker may be able to execute arbitrary code, alter the intended control flow, read sensitive information, or cause the system to crash.
September 14, 2016 OMG Cyber Risk Summit
©2016 Lockheed Martin Aeronautics Company
National Security Agency CAS Studies (2010 Report)
Chart 6
BlackHat 2011 Kris Britton https://www.youtube.com/watch?v=g0UL2Nam5hE
September 14, 2016 OMG Cyber Risk Summit
©2016 Lockheed Martin Aeronautics Company
NSA Data
Presented by NSA at OMG Meeting in DC, Mar 2012
September 14, 2016 OMG Cyber Risk Summit Chart 7
©2016 Lockheed Martin Aeronautics Company
NSA Reported Coverage Chart C/C++ Weakness Classes
September 14, 2016 OMG Cyber Risk Summit Chart 8
©2016 Lockheed Martin Aeronautics Company
NSA Study Conclusions
September 14, 2016 OMG Cyber Risk Summit Chart 10
©2016 Lockheed Martin Aeronautics Company
Lockheed Martin Aeronautics Study
Chart 11 September 14, 2016 OMG Cyber Risk Summit
©2016 Lockheed Martin Aeronautics Company
Lockheed Martin Aeronautics SCA Study (2011)
• Current Tool Inefficiency (based on NIST test cases) − Tool D* − Tool E*
Chart 12
32%
68%
SCA Coverage C/C++ Tool D Undetected
23%
77%
SCA Coverage Java Tool E Undetected
C/C++ used on Air Vehicle Java used on Ground Systems
*Tools D and E are not the same as the NSA tools D / E. September 14, 2016 OMG Cyber Risk Summit
©2016 Lockheed Martin Aeronautics Company
C/C++ Coverage Lockheed Martin Study (2012)
Chart 13
Determine which 2 tools in combination provides broadest coverage September 14, 2016 OMG Cyber Risk Summit
©2016 Lockheed Martin Aeronautics Company
2012 SCA Tool Study Results
• Commercial Off-the-Shelf (COTS) − Tool A* − Tool B* − Tool C*
• Tool A and Tool C provided 60.2% coverage of the C/C++ Juliet test cases − This particular pairing of the 2 provided the best coverage.
Chart 14
*Tools A, B, and C are not the same as the NSA tools A, B, C. September 14, 2016 OMG Cyber Risk Summit
©2016 Lockheed Martin Aeronautics Company
Coverage based on 2012 COTS Study (60.2%)
Chart 15
Tool A Tool C 15
23
36
114 134
190
195
197
222 223 252
226
242
244
247
253
256
259 272
304
319
321
327
328
338
364
367
369
390
396
400 401
404 415
416
457
459
467
468 476
478
480
481
482
483
489
500
506
534
562
563
570
571
590
591
605
606
617
672
675
676
680
685
688 690
758 761
762
773
789
835
843
121
122
123
124
126
127
188
191
204
273
284
325
374
377
391
392
398
440
469
475
484
78
194
196
426
427
464
479
510
511
535
546
547
560
561
587
588
620
665
666
674
775
780
785
832
366
The CWE ID’s shown are those in the Juliet test cases. September 14, 2016 OMG Cyber Risk Summit
©2016 Lockheed Martin Aeronautics Company
Tool A
Tool B coverage / non coverage
Chart 16
Tool C 15
23
36
114 134
190
195
197
222 223 252
226
242
244
247
253
256
259 272
304
319
321
327
328
338
364
367
369
390
396
400 401
404 415
416
457
459
467
468 476
478
480
481
482
483
489
500
506
534
562
563
570
571
590
591
605
606
617
672
675
676
680
685
688 690
758 761
762
773
789
835
843
121
122
123
124
126
127
188
191
204
273
284
325
374
377
391
392
398
440
469
475
484
78
194
196
426
427
464
479
510
511
535
546
547
560
561
587
588
620
665
666
674
775
780
785
832
Tool B found 6 additional, did not report 32 (False Negatives).
366
September 14, 2016 OMG Cyber Risk Summit
©2016 Lockheed Martin Aeronautics Company
Observations of Lockheed Martin 2012 SCA Study
• Each COTS product’s output was formatted differently − Analyzing the data was labor intensive!
• Developers would need to be trained on each individual tool • Combining the two COTS products gave slightly more coverage
− The COTS tools tend to look for the same weaknesses
Chart 17 September 14, 2016 OMG Cyber Risk Summit
©2016 Lockheed Martin Aeronautics Company
Approach (2013-14)
• Reviewed NSA Tool Capability Report, Static Analysis Tools for C, C++, and Java (March 24, 2011) (FOUO)
• Reviewed available SCA tool reports from NIST sponsored Static Analysis Tool Exposition (SATE) program
• Acquired Open Source Tool Output Integration Framework tool • Used validated C++ test cases
− NSA Juliet Test Suite v1.1
Chart 18 September 14, 2016 OMG Cyber Risk Summit
©2016 Lockheed Martin Aeronautics Company
Tool Output Integration Framework - TOIF
• TOIF initially developed in 2012 • in partnership of two companies (Data Access Technologies and
KDM Analytics) • Under DHS SBIR program with SBIR Topic Number: H-SB09.2-004
Software Testing and Vulnerability Analysis. • Goal: release TOIF technology as open source.
• Furthermore, KDM Analytics productized TOIF and make it available through: • open source • commercially available open source
− Maintenance updates, OSS License indemnification, trusted delivery
• commercially integrated with Blade Threat/Risk Analyzer (KDM Analytics proprietary solution)
Chart 19 September 14, 2016 OMG Cyber Risk Summit
©2016 Lockheed Martin Aeronautics Company
TOIF Architecture – High Level
Chart 20
TOIF Adaptor(s)
• CppCheck • Splint • Rats
…
TOIF Assimilator
Defect Generator Tool(s)
CppCheck Splint Rats
Eclipse (for SwEs)
KDM BLADE
(for SSEs)
TOIF Report View
• Open Source Tools (C/C++) included with TOIF − CppCheck 1.4 − Splint 3.1.2 − RATS 2.3
There are 2 Java Open Source Tools shipped with TOIF FindBugs and Jlint
September 14, 2016 OMG Cyber Risk Summit
©2016 Lockheed Martin Aeronautics Company
TOIF Flaw Reporting
Chart 21 September 14, 2016 OMG Cyber Risk Summit
©2016 Lockheed Martin Aeronautics Company
TOIF Coverage (NSA Weakness Classes)
Chart 22
0
2
4
6
8
10
12
14
16
18
20
TOIF Combined Coverage
3 Open Source SCATools
Total CWEs
September 14, 2016 OMG Cyber Risk Summit
©2016 Lockheed Martin Aeronautics Company
Cppcheck Splint
RATS
Coverage Based on TOIF Study (61.3%)
Chart 23
15
23
36
134
190
197
223 226
242
244
247
256
259
272
328
338
364
367
369
390
396
400
401
404
416
457
467
468
476
478
480
481
483
489
500
506
534
563
570
571
605
617
675
676
680
761
762
773 789
835 843
121
122
123
126
127
188
204
273
284
325
366
374
377
391
392
440
469
484
78
196
426
427
464
479
510
511
535
546
560
561
587
588
620
666 674
780
832
124 366
114
195
253
304
319 327
459
482
590
591 606
672
685
688 690
758
191
194
785
397
398
222
475
547
775
321
415
252
665
562
September 14, 2016 OMG Cyber Risk Summit
©2016 Lockheed Martin Aeronautics Company
TOIF Analysis
• TOIF provided 61.3% coverage of the C/C++ Juliet test cases • TOIF integrates the output from multiple SCA tools into an
Eclipse based view − TOIF significantly reduced the amount of time it takes review
the results from the SCA tools’ output. • TOIF uses Eclipse to display SCA tools output
− Eclipse tool is already in use at Aero
Chart 24
Additional adaptors could expand the CWE coverage.
• C/C++ − Sparse − Uno − BLAST − Fragma-C
• Java − Checkstyle − Sonar − PMD
September 14, 2016 OMG Cyber Risk Summit
©2016 Lockheed Martin Aeronautics Company
Benefits of TOIF
• TOIF has the flexibility of using Open Source SCA tools − Reduces acquisition/licensing cost of SCA tools − Reduces SCA training costs − Increases coverage of finding flaws / defects − Reduces manpower required in safety critical / security relevant
code reviews (focus on only defects not covered by TOIF) − Improve software quality by incorporating SCA tool analysis
during development phase • Simplifies the effort for the developer to incorporate flaw
remediation − Reduce software rework (hence costs) late in the development
process
Chart 25 September 14, 2016 OMG Cyber Risk Summit
©2016 Lockheed Martin Aeronautics Company
Conclusion
• TOIF is the better solution for performing Static Code Analysis to assist the developer − Single execution point − Normalized and standardized output format − Eclipse based – developers will not have to trained on another tool
• TOIF provides a mechanism to “tune out” false positives on individual adapted SCA tools.
• LM Aeronautics Company can add additional tests / tune existing ones in the OSS SCA tools to cover CWE’s relevant to Avionics.
• Adapt additional SCA tools to TOIF to expand CWE coverage for Developers, System Security Engineering teams and C&A efforts.
• Substantial reduction in licensing costs.
Chart 26
Specific capabilities of individual commercial tools should be licensed on a case by case SSE requirement.
September 14, 2016 OMG Cyber Risk Summit
©2016 Lockheed Martin Aeronautics Company
TOIF References
• Presentation on original TOIF project − http://www.dhs.gov/sites/default/files/publications/csd-edwin-
seidewitz-data-access-technologies.pdf • Additional DHS SBIR funding to enhance TOIF
− https://www.sbir.gov/sbirsearch/detail/402647 • KDM Site about TOIF
− http://www.kdmanalytics.com/toif/index.html
Chart 27 September 14, 2016 OMG Cyber Risk Summit
©2016 Lockheed Martin Aeronautics Company
Questions
Chart 28 September 14, 2016 OMG Cyber Risk Summit