Using ThreadFix to Manage Application Vulnerabilities

© Copyright 2013 Denim Group - All Rights Reserved

Using ThreadFix to Manage Application Vulnerabilities!!Dan Cornell!CTO, Denim Group!@danielcornell

© Copyright 2013 Denim Group - All Rights Reserved 2

My Background

•  Dan Cornell, founder and CTO of Denim Group

•  Software developer by background (Java, .NET, etc)

•  OWASP San Antonio, Global Membership Committee


Denim Group Background

•  Secure software services and products company –  Builds secure software –  Helps organizations assess and mitigate risk of in-house developed and third party

software –  Provides classroom training and e-Learning so clients can build software securely

•  Software-centric view of application security –  Application security experts are practicing developers –  Development pedigree translates to rapport with development managers –  Business impact: shorter time-to-fix application vulnerabilities

•  Culture of application security innovation and contribution –  Develops open source tools to help clients mature their software security programs

•  Remediation Resource Center, ThreadFix –  OWASP national leaders & regular speakers at RSA, SANS, OWASP, ISSA, CSI –  World class alliance partners accelerate innovation to solve client problems

3


Agenda •  Introductions •  Application Vulnerability Management •  ThreadFix Background •  Use Cases / Demonstrations

–  Track Scan Results Over Time –  De-Duplicate and Merge Multiple Scanners –  Scanner Benchmarking –  Virtual Patching –  Turning Vulnerabilities into Software Defects –  Program Benchmark Reporting

•  Future Directions •  Questions

4


Application Vulnerability Management

•  Application security teams uses automated static and dynamic test results as well as manual testing results to assess the security of an application

•  Each test delivers results in different formats

•  Different test platforms describe same flaws differently, creating duplicates

•  Security teams end up using spreadsheets to keep track manually

•  It is extremely difficult to prioritize the severity of flaws as a result

•  Software development teams receive unmanageable reports and only a small portion of the flaws get fixed

5


The Result •  Application vulnerabilities persist in applications:

**Average serious vulnerabilities found per website per year is 79 **Average days website exposed to one serious vulnerability is 231 days **Overall percentage of serious vulnerabilities that are fixed annually is only 63%

•  Part of that problem is there is no easy way for the security team and application development teams to work together on these issues

•  Remediation quickly becomes an overwhelming project

•  Trending reports that track the number of reduced vulnerabilities are impossible to create

**WhiteHat Statistics Report (Summer 2012): https://www.whitehatsec.com/assets/WPstats_summer12_12th.pdf


Vulnerability Fun Facts: •  Average number of serious

vulnerabilities found per website per year is 79 **

•  Serious Vulnerabilities were fixed in ~38 days **

•  Percentage of serious vulnerabilities fixed annually is only 63% **

•  Average number of days a website is exposed, at least one serious vulnerability ~231 days

WhiteHat Statistics Report (Summer 2012): https://www.whitehatsec.com/assets/WPstats_summer12_12th.pdf


Vulnerability Remediation Data

Vulnerability Type Sample Count Average Fix (minutes) Dead Code (unused methods) 465 2.6 Poor logging: system output stream 83 2.9 Poor Error Handling: Empty catch block 180 6.8 Lack of AuthorizaKon check 61 6.9 Unsafe threading 301 8.5 ASP.NET non-‐serializable object in session 42 9.3 XSS (stored) 1023 9.6 Null Dereference 157 10.2 Missing Null Check 46 15.7 XSS (reflected) 25 16.2 Redundant null check 21 17.1 SQL injecKon 30 97.5

8


Where Is Time Being Spent?

9

17%

37%

20%

2%

24%

0%

15%

0% 0%

9%

31%

59%

44%

15%

42%

16%

29% 24%

3%

28%

0%

10%

20%

30%

40%

50%

60%

70%

Setup Development Environment

Fix Vulnerabilities Confirm Fixes / QA Deploy Overhead

Indicates the weighted average versus the average of individual projects


Enter ThreadFix •  An open source software vulnerability aggregation and management system

•  Imports dynamic, static and manual testing results into a centralized platform

•  Removes duplicate findings across all testing platforms to provide a prioritized list of security faults

•  Eases communication across development, security and QA teams

•  Exports the prioritized list into the company’s bug tracker of choice to streamline software remediation efforts

•  Auto generates web application firewall rules to protect corporate data while the software vulnerability is being fixed

•  Empowers managers with vulnerability trending reports that can pinpoint team issues and illustrate application security progress

10


ThreadFix Background •  An open source vulnerability management and aggregation platform that

allows software security teams to reduce the time it takes to fix software vulnerabilities

•  Freely available under the Mozilla Public License (MPL)

•  Download available at: www.denimgroup.com/threadfix

11


ThreadFix Consolidates reports so managers can speak intelligently about the status and trends of security within their organization


Vulnerability Import • Pulls in static and dynamic results • Eliminates duplicate results • Allows for results to be grouped



Real-Time Protection Virtual patching helps protect organizations during remediation



Defect Tracking Integration

• ThreadFix can connect to common defect trackers • Defects can be created for developers • Work can continue uninterrupted



Large Range of Tool Compatibility

19


Supported Tools: Dynamic Scanners Acunetix Arachni Burp Suite HP WebInspect IBM Security AppScan Mavituna Security Netsparker NTO Spider OWASP Zed Attack Proxy Tenable Nessus Skipfish w3aF Static Scanners FindBugs IBM Security AppScan Source HP Fortify SCA Microsoft CAT.NET Brakeman

20

SaaS Testing Platforms WhiteHat Veracode QualysGuard WAS 2.0 IDS/IPS and WAF DenyAll F5 Imperva mod_security Snort Defect Trackers Atlassian JIRA Microsoft Team Foundation Server Mozilla Bugzilla


Use Cases / Demonstrations •  Track Scan Results Over Time •  De-Duplicate and Merge Multiple Scanners •  Scanner Benchmarking •  Virtual Patching •  Turning Vulnerabilities into Software Defects •  Program Benchmark Reporting

21


Track Scan Results Over Time •  Pretty basic, but many software security programs have problems

providing even basic metrics and trending graphs •  Goal: Turn a “dude with a scanner” into a “dude with some data”

•  Notes: –  Each new scan is diff-ed against the previous scan –  Vulnerabilities are tracked as new, fixed, reopened –  You can durably mark false positives

22


Track Scan Results Over Time •  Demonstration

23


De-Duplicate and Merge Multiple Scanners •  Q: What’s worse than handing a developer a 300 page PDF? •  A: Handing a developer two 300 page PDFs!

•  Communicating vulnerabilities via PDF is a horrible interaction pattern for security and development teams (more on this later)

24


What is a Unique Vulnerability? •  (CWE, Relative URL)

–  Predictable resource location –  Directory listing misconfiguration

•  (CWE, Relative URL, Injection Point) –  SQL injection –  Cross-site Scripting (XSS)

•  Injection points –  Parameters – GET/POST –  Cookies –  Other headers

25


What Do The Scanner Results Look Like?

•  Usually XML –  Skipfish uses JSON and gets packaged as a ZIP

•  Scanners have different concepts of what a “vulnerability” is –  We normalize to the (CWE, location, [injection point]) noted before

•  Look at some example files

•  Several vendors have been really helpful adding additional data to their APIs and file formats to accommodate requests (thanks!)

26


Why Common Weakness Enumeration (CWE)? •  Every tool has their own “spin” on naming vulnerabilities

–  OWASP Top 10 / WASC XX are helpful but not comprehensive

•  We tried to create our own vulnerability classification scheme –  Proprietary –  Not sustainable –  Stupid

•  CWE is pretty exhaustive •  Reasonably well-adopted standard •  Many tools have mappings to CWE for their results

•  Main site: http://cwe.mitre.org/

27


Challenges Using the CWE •  It is pretty big (909 nodes, 693 actual weaknesses)

•  But it kind of has to be to be comprehensive…

•  Many tools provide mappings •  And sometimes they’re even kind of accurate!

•  Some tools provide more than one CWE category for a vulnerability •  So in ThreadFix we make a best guess

•  Some tools provide “junk” results •  So in ThreadFix we collapse those into a single vulnerability

•  Some organizations have their own classification schemes


De-Duplicate and Merge Multiple Scanners •  Demonstration

29


Scanner Benchmarking •  Of the scanning technologies you are using, which is providing the

most value?

30


Scanner Coverage •  You can’t test what you can’t see

•  How effective is the scanner’s crawler?

•  How are URLs mapped to functionality? •  RESTful •  Parameters

•  Possible issues: •  Login routines •  Multi-step processes •  Anti-CSRF protection

31


Are You Getting a Good Scan? •  Large financial firm: “Our 500 page website is secure because the

scanner did not find any vulnerabilities!”

•  Me: “Did you teach the scanner to log in so that it can see more than just the homepage?”

•  Large financial firm: “…”

32


Did I Get a Good Scan?

•  Scanner training is really important •  Read the Larry Suto reports…

•  Must sanity-check the results of your scans

•  What URLs were accessed? •  If only two URLs were accessed on a 500 page site, you probably have a bad scan •  If 5000 URLs were accessed on a five page site, you probably have a bad scan

•  What vulnerabilities were found and not found? •  Scan with no vulnerabilities – probably not a good scan •  Scan with excessive vulnerabilities – possibly a lot of false positives

33


Low False Positives

•  Reports of vulnerabilities that do not actually exist

•  How “touchy” is the scanner’s testing engine?

•  Why are they bad? –  Take time to manually review and filter out –  Can lead to wasted remediation time

34


Low False Negatives

•  Scanner failing to report vulnerabilities that do exist

•  How effective is the scanner’s testing engine?

•  Why are they bad? –  You are exposed to risks you do not know about –  You expect that the scanner would have found certain classes of vulnerabilities

•  What vulnerability classes do you think scanners will find?

35


Other Benchmarking Efforts

•  Larry Suto’s 2007 and 2010 reports •  Analyzing the Accuracy and Time Costs of Web Application Security Standards

–  http://ha.ckers.org/files/Accuracy_and_Time_Costs_of_Web_App_Scanners.pdf •  Vendor reactions were … varied

–  [Ofer Shezaf attended this talk at AppSecEU 2012 and had some great questions and comments. See his reactions to the latest Larry Suto scanner report here :http://www.xiom.com/2010/02/09/wafs-are-not-perfect-any-security-tool-perfect ]

•  Shay Chen’s Blog and Site •  http://sectooladdict.blogspot.com/ •  http://www.sectoolmarket.com/ •  http://www.infosecisland.com/blogview/21926-A-Step-by-Step-Guide-for-Choosing-the-Best-

Scanner.html

•  Web Application Vulnerability Scanner Evaluation Project (wavsep) •  http://code.google.com/p/wavsep/

36


Scanner Benchmarking •  Demonstration

37


Virtual Patching

38

•  Connect vulnerability scanners to IDS/IPS/WAF systems

•  Map data from sensors back to data about vulnerabilities


Virtual Patches - Formats •  Two approaches

1.  (vulnerability_type, vulnerability_location) 2.  (vulnerability_signature , vulnerability_location)

(1) “There is a reflected XSS vulnerability in login.php for the username parameter”

versus (2) “Watch out for HTML-ish characters in login.php for the username parameter”

•  The snort and mod_security rules follow approach (2) •  Integration with commercial solutions may use approach (1)

39


Trivia and Analysis •  IDS/IPS/WAF has an impact on the scanning process

–  Snort breaks w3af scanning –  mod_security CRS introduces some false positives into skipfish scanning

•  mod_security CRS is quite good –  And getting better all the time: SQL Injection Challenge –  http://blog.spiderlabs.com/2011/06/announcing-the-modsecurity-sql-injection-challenge.html

•  Virtual patching appears to win for injection flaws

40


Where Is This Useful? •  Environments where you have little or no control over deployed code

–  XaaS – PaaS, IaaS –  99% of all corporate data centers

•  Environments where you have a large “application security debt” –  Actual code fixes: take time and can be hard to get on the schedule

41


What Are The Problems? •  Current vulnerability data formats only allow for coarse-grained virtual

patches –  Can lead to false blocks

•  Virtual patches likely will not stop well-informed, determined attackers –  See the results of the mod_security SQL Injection Challenge

42


Virtual Patching •  Demonstration

43


Turning Vulnerabilities Into Software Defects •  Security teams talk about “vulnerabilities” •  Software developers talk about “defects”

•  Developers Don’t Speak PDF –  http://blog.denimgroup.com/denim_group/2012/11/hey-security-teams-developers-dont-speak-pdf.html

•  Why should developers manage 90% of their workload in defect trackers

–  And the magic, special “security” part of their workload … some other way?

•  ThreadFix lets you slice, dice and bundle vulnerabilities into software defects

–  And track their remediation status over time to schedule re-scans

44


But My Bug Tracker Isn’t Supported!

•  We are always working on supporting new technologies –  Check out the current support list:

https://code.google.com/p/threadfix/wiki/DefectTrackers –  Submit a bug to the TheadFix defect tracker

https://code.google.com/p/threadfix/issues/list

•  You can add new defect trackers as plugins –  No changes to the core codebase required –  For instructions and sample code check out the wiki article:

https://code.google.com/p/threadfix/wiki/CustomDefectTrackerGuide

45


Turning Vulnerabilities Into Software Defects •  Demonstration

46


Program Benchmark Reporting •  How does your software security organization stack up?

–  Look at publicly-shared data from WhiteHat and Veracode

•  Compare your progress –  Percentage of vulnerabilities fixed –  Time to fix different vulnerability types –  Age of remaining vulnerabilities

47


Program Benchmark Reporting •  Demonstration

48


Current Status •  1.0 released September 17th, 2012 •  1.0.1 released October 19th, 2012 •  1.1 (release candidate) released January 28th, 2013 •  Final 1.1 coming in the next couple of weeks

49


Future Directions •  Increase the audience that can find ThreadFix useful

–  Add native scanning capability –  Add scan scheduling and coordination capability

•  Address “enterprise” concerns –  Expanded security model available in version 1.1 –  Continue to grow this area

•  Improve the user experience

•  Dashboard and reporting

50


Common Usage Scenarios •  Use ThreadFix to provide an “enterprise” console for a standalone

desktop scanning tool

•  Use ThreadFix to normalize and merge multiple sources of vulnerability data

–  Including the results of manual code reviews, threat models, etc

•  Use ThreadFix as a base for a custom application vulnerability management solution

–  We’ve already written a LOT of code and solved a LOT of problems

51


How Can You Help? •  Use it and provide feedback

–  Bug reports –  Usability recommendations –  Feature requests

•  Scan file examples –  Multiple tools, multiple versions, limited sample set –  Help!

•  Contribute

52


How To Get ThreadFix •  Denim Group ThreadFix homepage: www.denimgroup.com/threadfix

•  Google Code site: https://code.google.com/p/threadfix/

•  Google Group: https://groups.google.com/forum/?fromgroups#!forum/ThreadFix

53


Conclusions / Questions

Dan Cornell [email protected] Twitter: @danielcornell www.denimgroup.com www.denimgroup.com/threadfix code.google.com/p/threadfix (210) 572-4400

Technology

Using ThreadFix to Manage Application Vulnerabilities