Start with passing tests (tdd for bugs) v0.5 (22 sep 2016)

Start with passing Tests (TDD for bugs)

London, 22nd Sep 2016, v0.5

Me• Developer for 25 years

• AppSec for 13 years

• Day jobs:

• Leader OWASP O2 Platform project

• Application Security Training for JBI Training

• Part of AppSec team of:

• The Hut Group

• BBC

• AppSec Consultant and Mentor

• @Leanpub (buy for 0$ )

• http://leanpub.com/u/DinisCruz

–

Books Published

• published at Leanpub (buy for 0$ ) – http://leanpub.com/u/DinisCruz

Books under development

Major revision with lots of new content (based on Maturity Models app)

Ideas shown in this presentation and a lot more

See also:

http://blog.diniscruz.com/2016/03/new-era-of-software-with-modern.html

See also:

http://blog.diniscruz.com/2016/03/new-era-of-software-with-modern.html

TDD

• For bugs, always start with a passing test – Common workflow on TDD is to write failed tests.

– The problem with this approach is that it only works for a very specific scenario (when fixing bugs).

– This presentation will present a different workflow which will make the coding and testing of those much much easier, faster, simpler, secure and thorough'

Key concept: Turning TDD upside down

• For TDD to be productive you need – Real time unit test execution (when hands lift)

– Real time code coverage

• TDD focus needs to be on – making developers more productive

– preventing developers from switching context

• You need something like Wallaby or NCrunch

• If 99% code coverage doesn’t happen ‘by default’ TDD workflow is not working

TDD

9 9 % C O D E C O V E R A G E

…is not the destination

…it is ‘base camp’

TDD in WebStorm with WallabyJS

What happens when you increase attack surface

You want a test to fail

TDD in WebStorm with WallabyJS

• … but is a topic for another talk :)

TDD WORKFLOW FOR A BUG

• Bug is found

• Issue created and documented

• Issue prioritised and scheduled

• Fix

• QA

• Deploy – hope is has been fully fixed?

Normal flow for bugs and security issues

• Bug found, Issue created, Scheduled

• TDD loop – Write test that fails

– Fix Issue

– Test passes

– All done?

• QA – why is this needed?

With TDD

• Very few do this loop in pure TDD

– very hard to do without realtime test execution and code coverage

• Needs test API & Infrastructure to be there already

• are you able to spend an extra 4h, 8h,16h fixing a blind spot of your test APIs?

• End result tends to be very shallow tests

• Root cause of issue is usually not identified (or fixed)

• Given these bugs: • Login + open page X + enter data + submit data =

BUG happens

• Create object + populate with data + do action = BUG happens

• call service A + inner call to Service B + bad data returned from Service B = BUG happens

• If you write a failed test, what happens when a problem occurs before we reach the Bug happens

How do you know what failed?

• Tests are only written while in ‘fix mode’

• Very little time to improve test APIs for test execution environment

– There are really hard problems in Testing, that will need time to be solved • Creating test proxies

• Creating ‘hot’ run-time environments

• Improving test execution and code coverage visualisation (when Wallaby or NCrunch is not available)

• Will break the build

• Will create a high threshold of what can be tested

The problem with failed tests

• Missing Headers on Production (HSTS for example)

• Debug messages shown on production

• SQL Injection on user controller field

• Wrong product shown on page

• Wrong link generated on page and email

• Performance issues (for example: slow search)

• Inconsistent error messages returned from Web Service

• Message Queues accepting message with bad data

• Expose user info via parameter manipulation

• Over-posting on Controller’s Model bindings

• Use of Strings on models to hold data (‘strings should to be banned’)

Example of bugs

• Aren’t they the current behaviour of the application?

• Isn’t that how the application was designed

• If you are writing a test to replicate the issue, should that test: Pass or Fail?

• My definition of feature is: Once the code hits the main ‘develop’ branch, it’s behaviour is a feature (regardless if the code is in production)

Are these bugs or features?

Until you write the (passing) test, you don’t really know the root

cause of the issue

1.Find issue

2.Create PoC (Proof-of-Concept)

3.Report issue (on PDF or Bug tracking)

4.Dev ‘fixes’ issue

5.Fix pushed to production

6.Is it really fixed? If yes, end of story

7.Find variation of issue

8.go back to 2.

Bug/Security fix merry-go-round

• Why is QA finding issues that the devs are not aware of, or have been fixed before?

• Most QA teams are clicking on things and running scripts

• QA teams: – should be reviewing the Tests created/changed since the

last release

– need to have developers since they need to write Tests

– need to have senior developers since they need to improve the Test’s APIs and execution/development Environments

QA - Most are not being efficient

WRITING PASSING TESTS

• Write Test first

• Understand the issue

• Improve test APIs and Frameworks

• Write one Test per affected layer (or call stack) – start with e2e (end-to-end) and end with pure Unit Tests (on the root

cause of the issue)

– For affected each method, write tests for it’s callers (creating a reverse tree)

• By definition, the fix of original issue(s) will break these tests

• Transform these ‘passing bug tests’ into ‘passing regression tests’

• QA team (and manager/architect/security) reviews the test’s diff

Suggested workflow

• Team members need to communicate using Tests

• Tests should be easier to write than explanations (text, screenshot, videos)

• Teams collaboration should be done via tests (from dev to QA, from QA to dev, from Team A to team B)

• Managers need to write tests (and demand powerful/easy-to-use APIs)

Tests need to be the lingua franca

• It is crazy that we tend to promote the best developers into management position (since that is the ‘only’ career path) and then make them do everything except coding

• These ‘senior devs, now managers/architects’ tend to be the best programmers in the building and usually love coding

• The solution is to make them read and write tests

• Teams need to communicate using tests

• Printing tests (on paper) is a very effective way to code review

Managers need to code

When creating tests on the ‘Fix’ stage, the focus (& time allocated) is on fixing the bug (not on testing it)

When creating tests on the ‘Issue Creation’ stage, the focus (& time allocated) is on

how to test it and what is its root cause

CASE STUDY: WHEN I CREATED A VULNERABILITY

• Here is the code I wrote (at the Data Layer)

• This method is designed to be called by the controller (i.e. rest api endpoint):

Feature request: Allow data editing on UI

Feature request: Allow data editing on UI

Regression test that passes on issue

Fix for Path transversal

Regression test

LET’S SEE HOW IT LOOKED IN THE CODE

…before the vuln is created

…when the vuln is created

… adding comments

…after issues are created

…improving comments

…updating issues after 1st fix

… after final fix

Thanks, any questions

@diniscruz

dinis.cruz@owasp.org