Better Security Testing: Using the Cloud and Continuous Delivery

T17 Security Testing

5/2/2013 1:30:00 PM

Better Security Testing: Using the

Cloud and Continuous Delivery

Presented by:

Gene Gotimer

Coveros, Inc.

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Gene Gotimer

Gene Gotimer is a senior architect at Coveros, Inc., a consulting company that uses agile methods to accelerate the delivery of secure, reliable software. Gene is an experienced software developer who focuses on continuous integration, static code analysis, automation, and any tool he can find to do his work for him. For the past few years, he has been on a team that is bringing a continuous delivery process into the US Department of Defense, showing that higher quality software can be delivered quicker and with more security by using agile techniques.

4/16/2013

1

1© Copyright 2013 Coveros, Inc. All rights reserved.

Gene Gotimer, Senior Architect

[email protected]


� Coveros helps organizations accelerate the delivery of

business value through secure, reliable software

About Coveros

4/16/2013

2


Security TestingSecurity Testing

� Late in the cycle

� Issues are not remediated

� Needs ROI: lower cost, better results, or both


Security TestingSecurity Testing

� Consider open-source and free tools

� Opportunities in Continuous Delivery and Cloud

Identify low effort opportunities using free

and open-source tools

4/16/2013

3


Continuous IntegrationContinuous Integration

� Merge work frequently

� Code commits to source control

� Unit tests run automatically

� No long integration cycle at the end

� Fix code when we find problems

� Build-Test-Commit cycle = rapid feedback


Continuous DeliveryContinuous Delivery

� Every build potentially releasable

� Release is a business decision

� Extrapolation of Continuous Integration

– Deploys

– Functional tests

– Load and performance tests

– Security tests

� Build-Test-Commit-Deploy-Test-Release cycle

4/16/2013

4


Cloud ComputingCloud Computing

� Can’t wait for long procurement

� Public or Private clouds

� Works well with Continuous Delivery

– Easy to deploy

– New environments whenever


Maturity Model for Security TestingMaturity Model for Security Testing

� Level 0: No Security Testing

� Level 1: Unit Testing and Static Analysis

� Level 2: Automated Deploys and Functional Testing

� Level 3: Automated Configuration Management

� Level 4: Cloud Deployments

� Level 5: Continuous Delivery

4/16/2013

5



�� Level 0: No Security TestingLevel 0: No Security Testing









�� Level 1: Unit Testing and Static AnalysisLevel 1: Unit Testing and Static Analysis





4/16/2013

6


Level 1: Unit Level 1: Unit Testing and Static AnalysisTesting and Static Analysis

� Unit Tests:

– Confidence to make changes

– Error handling

– General logic errors

– Bounds checking

– Edge conditions


Level 1: Unit Testing and Static AnalysisLevel 1: Unit Testing and Static Analysis

� Static Analysis:

– Common errors

– Unused variables

– SQL injection

– Cross-Site Scripting (XSS)

– Hard-coded passwords

4/16/2013

7


Level 1: Unit Testing and Static AnalysisLevel 1: Unit Testing and Static Analysis

� Unit testing:

� JUnit for Java

� NUnit for .Net

� PyUnit for Python

� PHPUnit for PHP

� Static Analysis:

� Sonar for many languages

� PMD for Java

� FindBugs for Java

� PHPMD for PHP

� FxCop for .Net

� PyChecker for Python

� pylint for Python





�� Level 2: Automated Deploys and Functional TestingLevel 2: Automated Deploys and Functional Testing




4/16/2013

8


Level 2: Automated Deploys and Functional Level 2: Automated Deploys and Functional TestingTesting

� Automated Deploys:

– Frequent security scans

– Rapid feedback

– Web application scanners:

� w3af

� wapiti

� Skipfish

– Start basic, add tuning


Level 2: Automated Deploys and Functional TestingLevel 2: Automated Deploys and Functional Testing

� Functional Testing:

– Access control

– Data protection

– Web Application testing:

� Selenium

4/16/2013

9


Level 2: Automated Deploys and Functional TestingLevel 2: Automated Deploys and Functional Testing

� Proxies:

– Better coverage

– XSS and Cross-Site Request Forgery (XSRF)

– Data leakage

– URLs for logs to augment spidering

– Web application proxies:

� OWASP Zed Attack Proxy (ZAP) Project

� OWASP WebScarab

� Ratproxy






�� Level 3: Automated Configuration ManagementLevel 3: Automated Configuration Management



4/16/2013

10


Level 3: Automated Configuration ManagementLevel 3: Automated Configuration Management

� Deployment/Configuration:

� Puppet

� Chef

� Provisioning:

� Cobbler

� Kickstart

� Windows Deployment Services

� Completely new systems or build on templates

� Repeatable configuration management


Level 3: Automated Configuration ManagementLevel 3: Automated Configuration Management

� Complete system scans

� OpenVAS

� Nmap

� Nikto2

4/16/2013

11







�� Level 4: Cloud DeploymentsLevel 4: Cloud Deployments



Level 4: Cloud Level 4: Cloud DeploymentsDeployments

� On-demand environments

– Long running scans in parallel

– Production-sized machines, even temporarily

– Failover and high-availability

� Multiple client systems in parallel

– Race conditions

– Multi-user interactions

4/16/2013

12


Level 4: Cloud DeploymentsLevel 4: Cloud Deployments

� Web performance testing frameworks:

� Apache Jmeter: Java-based UI

• HTTP, HTTPS, SOAP, JDBC, LDAP, JMS, SMTP, POP, IMAP

� ab, ApacheBench: command-line

� The Grinder: Jython and Clojure

� Gatling: Scala








�� Level 5: Continuous DeliveryLevel 5: Continuous Delivery

4/16/2013

13


Level 5: Continuous Level 5: Continuous DeliveryDelivery

� Release ready for production

� Continuous deployment

� High levels of automation

� Dashboards

– Custom development


Personal ExperiencePersonal Experience

� Agile development grew into Continuous Delivery

� Automated deploys with Puppet

� Selenium functional tests

� JMeter performance tests

4/16/2013

14



� Security testing lagged

� Excuses:

– The “official” tool is expensive.

– It would take a lot of time to acquire and then to configure it.

– We don’t have time.

– It isn’t our responsibility.

– The security team wouldn’t accept our scans anyway.

� Open-source tools

� Focus on security, not compliance

� Limited time



� Web application scans with w3af

� Vulnerability assessments with OpenVAS

� Security standards checks with Openscap

� Initial implementation ~ a day each

� No more freebies

4/16/2013

15


ConclusionConclusion

� Earlier security testing

– Less likely to skip

– More likely to remediate

� Open-source tools

� Other testing as foundation

� Gradually add more security tests

� Continuous Delivery and Cloud Computing give security testing opportunities


Questions?Questions?

1

Better Security Testing Using the Cloud and Continuous Delivery Gene Gotimer Coveros, Inc.

Security testing is often put off as one of the last tasks of the development cycle. So when the security scans or reviews find vulnerabilities, businesses can be put in the position of remediating the security issues appropriately and delaying a release or keeping a release on schedule and accepting the risks. Those risks are not always clearly understood and almost never have a clear dollar figure associated with them. Without really knowing what they are getting into, security remediation may be dropped and the security testing treated as a checkbox that “we ran the security test tool” with no action being taken as a result.

It is easy to see how security testing can become an afterthought or a second-class citizen during the development process. It becomes a question of return on investment (ROI). To make a stronger case for additional security testing we either have to increase the return or decrease the investment.

There is good news. Open-source and free security testing tools are growing in popularity and capability. A development team can quickly (and, if needed, quietly) add security testing tools to their process, without procuring an expensive security tool. That reduces the cost of the initial investment and may eliminate some barriers to entry.

And it turns out that Cloud initiatives and efforts towards Continuous Delivery (CD) not only have some great benefits of their own, but can also provide some opportunities to get security testing in place without a large outlay of resources. For a team that is trying to increase their security testing as cheaply as possible, there are plenty of chances to add it to their process almost for free.

Those will be the focus of my discussion: low effort opportunities to integrate security testing into the development process, and the free and open-source tools that can make it practical.

While I will focus on free and open-source tools, commercial tools can be better and easier to use. A more mature security testing practice may choose to use the commercial tool instead. Commercial tools may already be available in your organization, which can ease adoption by being already paid for, by being easier to use, by being more capable, and especially by having some people in your organization that can help you use the tool and interpret the results.

Continuous Integration Continuous integration (CI) is the practice of merging the work from multiple developers frequently, often multiple times per day. It almost always involves frequent code commits to a source control system and running unit tests on each and every commit. The primary benefit is that there is no long

2

cycle of integration at the end of the development cycle with developers scrambling to make all our code work together. Plus, as the unit tests and the integration efforts find problems, we make adjustments to our code while it is fresh in the developer’s minds. Developers get into a build-test-commit cycle which gives us rapid feedback and makes working, unit-tested code available to the rest of the team on a continuous basis.

Continuous Delivery Continuous delivery is a software development process that aims to have every build releasable to production. It may not be released, but everyone involved will be able to make an informed business decision to release that software without having to guess whether or not it will work.

Continuous delivery can be viewed as the logical extrapolation of continuous integration. After running the normal unit tests on the code, we frequently deploy code and run other tests, such as functional tests, load and performance tests, and even security tests. By extending the CI concept of build-test-commit to be build-test-commit-deploy-test-release, we can extend the rapid feedback cycle all the way out to production releases, or at least to the point of releasable software. And instead of just having unit-tested code available to the other developers, now we have production-ready code available to the business.

Enabling continuous delivery means adding automation to deploys, so that deploys can happen without a lot of effort. If it is expensive in terms of time and effort to deploy, it won’t be done often. And if it isn’t automated, it won’t be repeatable and consistent. Not only do deploys to our test environments become non-events, but we know when we push into production that the deploy process will work the same way. After all, the deploy process has been tested just as often as the code has.

Cloud Computing Cloud computing has brought a new economy to acquiring and provisioning computing resources. Along with that, the cloud has brought a new wave of tools to make configuration management and software deployment capable of exploiting that new found speed and low costs.

Because we are deploying on a rapid schedule, we can’t wait for the normal weeks-to-months long procurement and installation cycles to get hardware ready for us to deploy to. With cloud computing, the procurement cycle can be minutes long. And the investment per project is usually lower, too.

If we use a public cloud like Amazon, cloud services are always available for ordering. Virtually any amount of computing power can be ordered whenever we need it. Private cloud computing resources are often pooled, so when we are done using them the resources can be returned to the pool for other teams to use.

Cloud computing has a symbiotic relationship with continuous delivery. Deploys are easy because they are automated, so we have less of a barrier to standing up new environments. And establishing new

3

environments in the cloud is easy, so we can stand one up from scratch whenever it would be convenient to have another instance of the application deployed.

Maturity Model for Security Testing To grow a security testing practice, we need to first take stock of where we stand now. Then we need to have some path to follow, milestones to shoot for, and some way to measure if we are making improvements or not. One way to measure our current state and chart our growth is a maturity model.

We’ll start by setting the scale. The levels might be good for comparing the maturity of security testing between two different projects, but they are really just intended as an example of milestones to aim for. Each level represents an opportunity to introduce new security tests and security testing tools into the process. The progression ranges from introducing security testing to an established practice and onto advancing and refining your security testing capabilities.

• Level 0: No Security Testing • Level 1: Unit Testing and Static Analysis • Level 2: Automated Deploys and Functional Testing • Level 3: Automated Configuration Management • Level 4: Cloud Deployments • Level 5: Continuous Delivery

These levels aren’t equally spaced in time or effort to achieve them. Going from nothing to unit tests and static analysis is much easier than going from automated configuration management to cloud deployments. In general, the more mature the practice, the more effort it will require to incorporate the next practice. The early ones are the types of testing we can add almost for free, while the later practices require a lot of discipline and a committed effort.

Largely for this reason, not every project will benefit from working towards the ultimate level. It might be sufficient to unit test and do static analysis on an internal utility with a limited lifespan and no sensitive data or access, while an online ordering system available to the public certainly needs more security attention. What constitutes “enough” security testing has to be determined for each project.

Level 0: No Security Testing The initial state is no security testing. We’ll score that as a zero. Unfortunately, far too many projects start and end here. The good news is that if we start here, even the slightest bit of improvement is a win.

Level 1: Unit Testing and Static Analysis The next step, barely doing any security testing, is very easy to integrate into our development process: simply add unit testing and static analysis. You might not normally file those under security tests, but they are a good foot in the door and if you are doing CI you probably have these already.

4

Perhaps the biggest advantage of having good unit tests is the confidence that we can make changes to an application safely, without making unintended changes to behavior. That decreases the risks involved with any code changes, including security remediation, even late in the development cycle. This makes unit testing a critical practice as the foundation for any development.

Through normal efforts in unit testing, most developers will start testing edge cases and failure cases. For example,

• What happens if we have no items in the list we are iterating through? • What happens when we expect 5 items but only get 4? • How do we handle a file not found when we try to read a configuration file? • What if the file is found but not in the correct format?

Poor handling of these types of errors can lead to some common type of security vulnerabilities: error handling, general logic errors, bounds checking, etc. Unit testing is well-suited to identifying some of these error types since the units are tested independently. Fewer assumptions about the input and program state are made, so the units are able to better stand on their own. As a result, it can be easier to test each unit more thoroughly for undesirable or unexpected conditions. Adding a code coverage tool can help us identify edge conditions and testing opportunities that we missed initially.

Static analysis tools look at source code or compiled code, looking for common errors, unused variables, style and formatting variations, and similar items. Like unit tests, static analysis tools can identify some potential security problems:

• Are inputs validated? • Are outputs escaped? • Is the source code hard to understand, and therefore easy for other developers to misuse? • Is there a lot of duplicated source code? If we fix a bug once will the same problem crop up

in other places?

Most modern languages have a selection of open-source static analysis tools that can scan source code looking for predictable problems. SQL injection, Cross-Site Scripting (XSS), and hard-coded passwords are common vulnerabilities that can often be detected by static code analysis.

With a CI practice, it is easy to configure unit testing and static code analysis to run on every commit or at least every night. We can make it part of a project’s boilerplate build scripts and/or even enable it in our CI engine with little more than the click of a configuration checkbox.

Just adding unit testing or static analysis isn’t a comprehensive security testing process, but it is easy to get started and is far better than nothing. And we need to have these anyway.

Here are some suggestions for some free and/or open-source tools to get started:

5

• Unit testing: JUnit for Java, http://junit.org NUnit for .Net, http://www.nunit.org PyUnit for Python, http://pyunit.sourceforge.net PHPUnit for PHP, http://www.phpunit.de

• Static Analysis: Sonar for many languages, http://www.sonarsource.org PMD for Java, http://pmd.sourceforge.net FindBugs for Java, http://findbugs.sourceforge.net PHPMD for PHP, http://phpmd.org FxCop for .Net, http://msdn.microsoft.com/en-us/library/bb429476.aspx PyChecker for Python, http://pychecker.sourceforge.net pylint for Python, http://www.pylint.org

Level 2: Automated Deploys and Functional Testing Automated deploys are the first step most teams think of when they start working towards continuous delivery. Automated deploys also open up some opportunities for more focused security testing.

The most obvious opportunity is just being able to scan the deployed application on a frequent basis. If we can deploy automatically, then we can kick off security tests after we deploy and get a scan early in the development cycle. We can investigate and remediate vulnerabilities we find before it is too late to address them.

Even better, if we can trigger a deploy and scan using our CI engine, then developers can get same-day or next-day feedback that shows that they may have introduced a security issue. They can fix problems while the code is still fresh on their minds.

The easiest way to start scanning an automated deploy is with black-box tools that can just be pointed at the deployed application and need little-to-no configuration initially. Some examples for web applications include:

• w3af, http://w3af.org • wapiti, http://wapiti.sourceforge.net • Skipfish, http://code.google.com/p/skipfish/

While these tools can (and should) eventually be tuned to better suit our application, even the most basic scans can provide a wealth of information. Simple tuning can usually limit runtimes to be short enough to fit the situation. For example, every hour we could run a quick, superficial scan that takes 10 minutes and then overnight we could run a more detailed scan.

Automated deployments also ease automated functional testing. In addition to the normal user behavior tests, we can use functional tests to test access controls and data protection. User roles can be very thoroughly tested using automated functional tests, especially when doing negative testing. For

http://junit.org/

http://www.nunit.org/

http://pyunit.sourceforge.net/

http://www.phpunit.de/

http://www.sonarsource.org/

http://pmd.sourceforge.net/

http://findbugs.sourceforge.net/

http://phpmd.org/

http://msdn.microsoft.com/en-us/library/bb429476.aspx

http://pychecker.sourceforge.net/

http://www.pylint.org/

http://w3af.org/

http://wapiti.sourceforge.net/

http://code.google.com/p/skipfish/

6

example, a user should be able to see their account information, but not someone else’s. Users with one role should be able to access certain functionality, but not other functionality.

The most popular open source functional test framework for web applications is Selenium (http://seleniumhq.org), although there are plenty of good alternatives. There are also many good options for applications that aren’t web-based or browser-based, such as web services.

Having those tests automated means they can be run frequently, and they become a great shield against regression bugs and unintended data leakage. Like unit tests, the strongest advantage of automated functional tests is that changes can be made to individual features without introducing unwanted behavior changes. If we enable a role to view a new aggregate report, did those users also get the ability to view someone else data? If we explicitly add the ability for a role to see another user’s data, are they still prohibited from making changes to it?

If we combine the automated functional tests with security scanning we get some very good insight into possible vulnerabilities. Running the functional tests through a proxy can uncover XSS and Cross-Site Request Forgery (XSRF) issues that a typical web scanner could not reach without login credentials or complex combinations of actions, for example. Proxies can also help find unintended external data leakage, such as a finding that a third-party library uses a public web service to generate charts. Collecting URLs from web access logs that were accessed during functional tests can be a great addition to spidering targets for web application scanners.

Free web application proxies include:

• OWASP Zed Attack Proxy (ZAP) Project, https://www.owasp.org/index.php/OWASP_Zed_Attack_Proxy_Project

• OWASP WebScarab, https://www.owasp.org/index.php/Category:OWASP_WebScarab_Project

• Ratproxy, http://code.google.com/p/ratproxy/

Level 3: Automated Configuration Management Configuration management means different things to different people, but I am referring to being able to determine and recreate the state of a computer system: OS, configuration files and settings, package versions, dependency versions, etc.

Frameworks such as Puppet (https://puppetlabs.com/puppet/puppet-open-source/) and Chef (http://www.opscode.com/chef/) are gaining a lot of traction for handling automated configuration management functions. They allow package versions, file permissions and ownership, and network services to be specified and controlled in code, rather than a list of manual steps a system administrator must remember to follow upon installation.

Puppet and Chef can be used alongside tools such as Cobbler (http://cobbler.github.com/), Kickstart (http://fedoraproject.org/wiki/Anaconda/Kickstart), or Windows Deployment Services (http://msdn.microsoft.com/en-us/library/aa967394.aspx) to stand up a system from “bare metal”. Or

http://seleniumhq.org/

https://www.owasp.org/index.php/OWASP_Zed_Attack_Proxy_Project

https://www.owasp.org/index.php/Category:OWASP_WebScarab_Project

http://code.google.com/p/ratproxy/

https://puppetlabs.com/puppet/puppet-open-source/

http://www.opscode.com/chef/

http://cobbler.github.com/

http://fedoraproject.org/wiki/Anaconda/Kickstart

http://msdn.microsoft.com/en-us/library/aa967394.aspx

7

they can be used in conjunction with a “Gold Disk” or other standardized OS installs to completely recreate a production-like environment and system laydown.

From a security point of view, the configuration management itself is the biggest draw. Being able to review and specify a system configuration via code and knowing that it represents the actual system state rather than the requested system state is invaluable during security audits. And the ability to recreate an environment has tremendous forensic value when troubleshooting a security incident, not to mention simply being able to reliably recreate production bugs in non-production environments.

Another advantage is that we can baseline (return to a known state) a complete system easily. That means that we can run a security scan on a cleanly installed system, and the entire system can be analyzed. Scanners like OpenVAS (http://www.openvas.org/), an open-source Nessus fork, Nmap (http://nmap.org/), and Nikto2 (http://cirt.net/nikto2) can be used against a system to scan for vulnerabilities via open ports and exposed services. We can schedule these tools to run regularly or scripted to run from a CI engine. Rather than wait for a security expert to schedule a scan of our application on a pre-production system just before release, we can scan the system every day so that we know the results and can remediate issues during the development cycle.

Level 4: Cloud Deployment Deploying our application in the cloud allows environments to be stood up without much time, effort, or cost. Since we’ve already automated the deployment and the OS and system installation, now standing up a new environment from nothing can be effortless, even without provisioned hardware.

That in turn means we can stand up an environment whenever we need it and only keep it operating while it is in use. More projects can share fewer resources since they won’t all need the resources at the same time. We can run long running scans, such as deep fuzzing scans, against a newly provisioned, dedicated environment while other development efforts continue in other environments in parallel.

Also, we can execute resource intensive tests with adequate resources even if those resources cannot be reserved indefinitely. Performance, load, and stress tests can be run on “production-sized” machines rather than possibly undersized development systems. That can help better identify limits and reduce the risks of denial of service attacks. Failover and high-availability systems often get tested once prior to the initial production release, and then never tested again, only because it was too expensive to have an entire extra set of redundant systems stood up in a test environment.

Some tests can benefit from the use of multiple client systems as well. Rather than running a many hours of functional tests overnight and having a one-day lag between code commit and test results, we can use cloud resources to stand up multiple client systems to run the functional tests in parallel. Not only can we run the tests more often since they take less elapsed time, but we’ll be better able to identify RACE conditions and multi-user interactions.

Open-source performance testing frameworks for web-based applications include:

http://www.openvas.org/

http://nmap.org/

http://cirt.net/nikto2

8

• Apache JMeter (http://jmeter.apache.org), which builds tests from a Java-based UI and can test HTTP, HTTPS, SOAP, JDBC, LDAP, JMS, SMTP, POP, and IMAP, among other protocols

• ab, ApacheBench (http://httpd.apache.org/docs/2.4/programs/ab.html), for command-line driven performance tests

• The Grinder (http://grinder.sourceforge.net), for load testing with tests written in Jython and Clojure

• Gatling (http://gatling-tool.org), for stress testing with tests written in Scala

Level 5: Continuous Delivery Continuous delivery means being able to delivery software at any time. It takes releasing software from what had been a lengthy and often risky development process to an automated process that ensure the timing of a release is a purely business decision. It can range in practice from having a release always ready to go should the stakeholder want to promote it to production, to continuous deployment where the software is tested thoroughly enough that stakeholders are confident that it can be promoted automatically, without any human decision points, whenever all automated testing reaches a “good enough” threshold.

This stage requires a very high level of automation, so that we have enough confidence in all testing (functional, regression, performance, and security testing) to allow unattended, automated releases. Most software, and certainly most organizations, will never reach this level of trust in automated testing, but it is a valuable goal.

Along with the automation itself, dashboards are important to display the results of all the various builds and tests through the process. The more automation and testing that is involved, the more information that has to be processed, and the more important dashboards become. The proliferation of environments makes it critical for us to be able to quickly determine what the configuration of each environment is, what packages are installed, what tests have been performed, and what the results of those tests were.

Unfortunately, there aren’t any ready-to-use open-source dashboards for a continuous delivery process, which makes sense since the process and tools can vary significantly from team to team and project to project. We will have to develop these dashboards as they are needed.

Personal Experience My current project is a large integration and customization project for the Department of Defense (DoD). As part of our mission, our team was tasked to introduce an agile development process and demonstrate how it could be used on a DoD project. That eventually became an effort to work towards a Continuous Delivery process, and to show if and how it would work in the DoD environment.

We’ve made great progress. Over the last year we have added deploys that are almost entirely automated using Puppet, although a few manual steps still remain. We have developed a suite of functional tests to cover our entire system. The system is web based, and we have chosen Selenium for

http://jmeter.apache.org/

http://httpd.apache.org/docs/2.4/programs/ab.html

http://grinder.sourceforge.net/

http://gatling-tool.org/

9

our functional tests. We have just recently added JMeter testing to measure performance, really focused on watching the trends as we make releases. I.e., have we made the system noticeably faster/slower?

One item we had not addressed until well into the project was security testing. We had done it a few times manually, but another part of the organization was responsible for the security scanning, so we didn’t build it into our process. If that sounds like a weak justification, it is. We just kept making excuses, such as:

• The “official” tool is expensive. • It would take a lot of time to acquire and then to configure it. • We don’t have time. • It isn’t our responsibility. • The security team wouldn’t accept our scans anyway.

While there is a grain of truth to each of those, we finally gave it some serious thought and realized that we already had the infrastructure and process to support some security testing, and could start doing a lot more with very little effort.

We decided we would focus on open-source tools that paralleled what the security team would eventually use. The more testing we did up front, the less likely we were to be surprised by findings at the end of the cycle. Focusing on improving the security was more important for our development team than worrying if we could satisfy compliance requirements by running particular tools. But since security truly was the responsibility of another team, we did not have any significant time to implement our testing.

We use w3af to do web application scanning. Our application relies on client-side certificates for login, so we had to make some slight customizations to w3af to deal with passing a certificate in. Fortunately, w3af is open source and written in Python, and it was easy to add the required support with just a few lines of code. We can run w3af from our CI engine, Jenkins, but don’t often because we haven’t come up with a convenient way to display the results. Since a manual inspection of the scan results is still required we don’t execute the w3af scans continuously. We are working on better reporting so we can quickly see what changes in the scans were made with the recent changes to the code.

We set up OpenVAS to do remote scans and vulnerability assessments. The scans run each weekend against our test systems. Ten servers are scanned in parallel; the scans are completed in less than two hours. We could scan more often, but the systems themselves don’t change open ports and installed packages often enough to warrant it. If needed, we can start the scans on demand. OpenVAS maintains a history of the scans and can track issues we’ve identified as false positives. It also shows changes and overall trends in the number of vulnerabilities.

We added Openscap (http://www.open-scap.org) as part of our CI process. It runs a series of automated checks against our systems to determine compliance with a set of security standards, which in our case are mandated. Security Content Automation Protocol (SCAP) has a fair number of checks that must be

http://www.open-scap.org/

10

manually performed as well, but at least running the automated checks makes our compliance efforts easier. Most importantly, we can see when a change to our system or to our application introduces or fixes any security vulnerabilities.

Each of these security scans was initially implemented in a day or so, using down time in between other assigned tasks. We added them to the process because they were open source, easy to introduce, and did not slow down or complicate our existing process. Also, it was easy to interpret the results, so we could act upon the results so that the scans had real value in our process. If you can’t understand the findings, you can’t act on them.

On our scale of maturity, we’ve pushed through automated configuration management and are beginning to do more cloud deployments, but our security testing is lagging a little behind that. We are just starting to take advantage of the cloud for performance testing and slower running security scans. On the other hand, we have a good selection of scans going on that help us ensure that our systems and our application are more secure and more likely to meet compliance rules. Also, we find out quickly if changes we make to the system configuration or to the application introduce any new security issues.

It seems like we’ve exhausted our “free” opportunities to increase the security testing. Most of what we’ll add in the future will require some real focus and effort. But the groundwork we have laid is already showing value, so garnering support for those larger efforts should be easier.

Conclusion Doing security testing earlier in the development cycle means it is less likely to be dropped and more likely we will remediate the problems that we find. By using open-source tools and identifying opportunities to add security testing into our current processes, we can reduce the cost of adopting security testing. Other, non-security testing can lay the foundation for security tests and safe remediation. Gradually adding security testing, even little pieces of it, can quickly add value.

Whether you already have continuous delivery and cloud initiatives underway and are piggybacking on them, or are using additional security testing potential as an argument to adopt them, the continuous delivery process and cloud computing efforts give ample opportunities to add security testing without much effort.

About the Author Gene Gotimer is a senior architect at Coveros, Inc., a consulting company that uses agile methods to accelerate the delivery of secure, reliable software. Gene is an experienced software developer who focuses on continuous integration, static code analysis, automation, and any tool he can get to do his work for him. For the last few years he has been on a team that is bringing a continuous delivery process into the US Department of Defense, showing other projects that software can be delivered quicker with more security and higher quality by using agile techniques.

Technology

Better Security Testing: Using the Cloud and Continuous Delivery