ITB2016 -BDD testing and automation from the trenches

BDD Testing & Automation from the Trenches Into the Box 2016

Gavin Pickin

★Who Am I?★Gavin Pickin – developing Web Apps since late 90s

○Ortus Solutions Software Consultant

○ContentBox Evangelist

★What else do you need to know?

○CFMLRepo.com http://www.cfmlrepo.com

○Blog - http://www.gpickin.com

○Twitter – http://twitter.com/gpickin

○Github - https://github.com/gpickin

★Lets get on with the show.

http://www.cfmlrepo.com

http://www.gpickin.com

http://twitter.com/gpickin

https://github.com/gpickin

★State of the Room

★ A few questions for you guys★ If you have arms, use them.

★State of the RoomTesting? What’s testing?


Yeah,

I’ve heard of it.

Why do you

think I’m here?

★State of the RoomYes I know I should be testing, but I’m not sure how to do it

★State of the RoomMy Boss and my Customers wouldn’t let me


I’m a tester

★State of the RoomI’m a test writing ninja

Call me Majano,

Luis Majano

★Ways to Test your Code★Click around in

the browser yourself

★Setup Selenium / Web Driver to click around for you

★Structured Programmatic Tests

★Types of Testing

★Types of Testing

★Black/White Box

★Unit Testing

★Integration Testing

★Functional Tests

★System Tests

★End to End Tests

★Sanity Testing

★Regression Test

★Acceptance Tests

★Load Testing

★Stress Test

★Performance Tests

★Usability Tests

★+ More

★Levels of Testing

★Cost of a Bug

The bug will cost one way or another



★Integration Tests several of the pieces

together

★Most of the types of tests are variations of an Integration Test

★Can include mocks but can full end to end tests including DB / APIs

★Unit Testing

★Unit Testing

“unit testing is a software verification and validation method in which a programmer tests if individual units of source code are fit for use. A unit is the smallest testable part of an application”- wikipedia

★Unit Testing★Can improve code quality -> quick error

discovery

★Code confidence via immediate verification

★Can expose high coupling

★Will encourage refactoring to produce > testable code

★Remember: Testing is all about behavior and expectations

★Styles – TDD vs BDD

★TDD = Test Driven Development

○Write Tests

○Run them and they Fail

○Write Functions to Fulfill the Tests

○Tests should pass

○Refactor in confidence

★Test focus on Functionality

★Styles – TDD vs BDD★BDD = Behavior Driven Development

Actually similar to TDD except:

○Focuses on Behavior and Specifications

○Specs (tests) are fluent and readable

○Readability makes them great for all levels of testing in the organization

★Hard to find TDD examples in JS that are not using BDD describe and it blocks

★TDD Example

Test( ‘Email address must not be blank’, function(){

notEqual(email, “”, "failed");

});

★BDD ExampleDescribe( ‘Email Address’, function(){

It(‘should not be blank’, function(){

expect(email).not.toBe(“”);

});

});

★Matchers

expect(true).toBe(true);




★Matchers

expect(true).not.toBe(true);





★Matcher Samples


expect(a).not.toBe(null);

expect(a).toEqual(12);

expect(message).toMatch(/bar/);

expect(message).toMatch("bar");

expect(message).not.toMatch(/quux/);

expect(a.foo).toBeDefined();

expect(a.bar).not.toBeDefined();

★CF Testing Tools

★MxUnit was the standard

★TestBox is the new standard

★Other options

★TestBoxTestBox is a next generation testing framework for ColdFusion (CFML) that is based on BDD (Behavior Driven Development) for providing a clean obvious syntax for writing tests.

It contains not only a testing framework, runner, assertions and expectations library but also ships with MockBox, A Mocking & Stubbing Framework,.

It also supports xUnit style of testing and MXUnit compatibilities.

★TestBox TDD Examplefunction testHelloWorld(){

$assert.includes( helloWorld(), ”world" );

}

★TestBox BDD Exampledescribe("Hello world function", function() {

it(”contains the word world", function() {

expect(helloWorld()).toContain("world");

});

});

★TestBox New BDD Examplefeature( "Box Size", function(){

describe( "In order to know what size box I need

As a distribution manager

I want to know the volume of the box", function(){

scenario( "Get box volume", function(){

given( "I have entered a width of 20

And a height of 30

And a depth of 40", function(){

when( "I run the calculation", function(){

then( "the result should be 24000", function(){

// call the method with the arguments and test the outcome

expect( myObject.myFunction(20,30,40) ).toBe( 24000 );

});

});

});

});

});

});

★Using Testing in your Workflow

★Using HTML Test Runners○Keep a Browser open

○F5 refresh tests

★Command Line Tests

★Run testbox – manual

○Run tests at the end of each section of work

★Run Grunt-Watch – automatic

○Runs testbox on every file change

○Grunt can run other tasks as well,

minification etc

★Testing in your IDE

★Browser Views

○Eclipse allows you to open files in web view – uses HTML Runner

★Run testbox/ Grunt / in IDE Console

○Fairly Easy to setup

○See Demo– Sublime Text 3 (if we have time)

★Installing Testbox★Install Testbox – Thanks to Commandbox

○box install testbox

★Decide how you want to run Testbox

★Create a runner.cfm<cfsetting showDebugOutput="false"><cfparam name="url.reporter" default="simple"><cfparam name="url.directory" default="tests.specs"><cfparam name="url.recurse" default="true" type="boolean"><cfparam name="url.bundles" default=""><cfparam name="url.labels" default="">

<cfinclude template="/testbox/system/runners/HTMLRunner.cfm" >

★Create a Test Suite// tests/specs/CFCTest.cfc

component extends="testbox.system.BaseSpec" {

function run() {

it( "will error with incorrect login", function(){

var oTest = new cfcs.userServiceRemote();

expect( oTest.login( '[email protected]', 'topsecret').result ).toBe('400');

});

}

}

★Create a 2nd Test Suite// tests/specs/APITest.cfccomponent extends="testbox.system.BaseSpec" { function run() { describe("userService API Login", function(){ it( "will error with incorrect login", function(){ var email = "[email protected]"; var password = "topsecret”; var result = ""; http url="http://127.0.0.1:8504/cfcs/userServiceRemote.cfc?method=login&email=#email#&password=#password#" result="result”;

expect( DeserializeJSON(result.filecontent).result ).toBe('400'); }); }); }}

★ Running Testbox with runner.cfm

*Install Testbox Runner – Thanks Sean Coyne

*npm install testbox-runner

*Install Grunt Shell

*npm install grunt-shell

*Add Grunt Configuration

★Running Testbox with Grunt Watch

★Install Testbox Runner – Thanks Sean Coyne

○npm install testbox-runner

★Install Grunt Shell

○npm install grunt-shell

★Add Grunt Configuration

★Adding TextBox Config 1

module.exports = function (grunt) {

grunt.loadNpmTasks('grunt-shell');

grunt.initConfig({ … })

}


Watch: {

…

cfml: {

files: [ "tests/*.cfc"],

tasks: [ "testbox" ]

}

}


shell: {

testbox: {

command: "./node_modules/testbox-runner/index.js --colors --runner http://127.0.0.1:53874/tests/runner.cfm --directory /tests/specs --recurse true”

}

}


grunt.registerTask("testbox", [ "shell:testbox" ]);

grunt.loadNpmTasks('grunt-contrib-jasmine');

grunt.loadNpmTasks('grunt-contrib-watch');

★GruntFile.js Gists

Simple Jasmine + Testbox Example

https://gist.github.com/gpickin/9fc82df3667eeb63c7e7




★Testbox output with Grunt

★Testbox Runner JSON

★Testbox has several runners, you have seen the

HTML one, this Runner uses the JSON runner and then formats it.

★http://127.0.0.1:53874/tests/runner.cfm?

reporter=json

http://www.testableapi.local.com:8504/tests/runner.cfm?reporter=JSON&directory=/tests/specs&recurse=true

http://127.0.0.1:53874/tests/runner.cfm?reporter=json



★Testbox Runner JSON{"totalSuites":3,"startTime":1465879026042,"bundleStats":[{"TOTALSUITES":1,"STARTTIME":1465879026042,"TOTALPASS":0,"TOTALDURATION":60,"TOTALSKIPPED":0,"TOTALFAIL":0,"TOTALSPECS":0,"PATH":"tests.specs.integration.api.BaseAPITest","ENDTIME":1465879026102,"DEBUGBUFFER":[],"TOTALERROR":0,"NAME":"tests.specs.integration.api.BaseAPITest","ID":"1DDCA037-FF86-4E9B-B62231A290304E9F","SUITESTATS":[{"STARTTIME":1465879026102,"TOTALPASS":0,"TOTALDURATION":0,"TOTALSKIPPED":0,"TOTALFAIL":0,"TOTALSPECS":0,"BUNDLEID":"1DDCA037-FF86-4E9B-B62231A290304E9F","STATUS":"Skipped","PARENTID":"","SPECSTATS":[],"ENDTIME":1465879026102,"TOTALERROR":0,"NAME":"tests.specs.integration.api.BaseAPITest","ID":"231E5335-408A-41A4-AFC0A1199989F05E","SUITESTATS":[]}],"GLOBALEXCEPTION":""},{"TOTALSUITES":2,"STARTTIME":1465879026102,"TOTALPASS":4,"TOTALDURATION":2395,"TOTALSKIPPED":0,"TOTALFAIL":0,"TOTALSPECS":4,"PATH":"tests.specs.integration.api.UsersAPITest","ENDTIME":1465879028497,"DEBUGBUFFER":[],"TOTALERROR":0,"NAME":"tests.specs.integration.api.UsersAPITest","ID":"8B3B6F5D-C7F2-4E77-B701A9F646061CB3","SUITESTATS":[{"STARTTIME":1465879026571,"TOTALPASS":1,"TOTALDURATION":472,"TOTALSKIPPED":0,"TOTALFAIL":0,"TOTALSPECS":1,"BUNDLEID":"8B3B6F5D-C7F2-4E77-B701A9F646061CB3","STATUS":"Passed","PARENTID":"","SPECSTATS":[{"ERROR":{},"STARTTIME":1465879026571,"TOTALDURATION":472,"FAILORIGIN":{},"STATUS":"Passed","SUITEID":"60F5FB8B-3E97-46C0-9165FDD893DF08B4","ENDTIME":1465879027043,"NAME":"Tests the ability to create a user","ID":"EBF1E06B-80A9-476E-9AA4889200FD48DC","FAILMESSAGE":""}],"ENDTIME":1465879027043,"TOTALERROR":0,"NAME":"CRUD API Methods and Retrieval","ID":"60F5FB8B-3E97-46C0-9165FDD893DF08B4","SUITESTATS":[]},{"STARTTIME":1465879027043,"TOTALPASS":3,"TOTALDURATION":1354,"TOTALSKIPPED":0,"TOTALFAIL":0,"TOTALSPECS":3,"BUNDLEID":"8B3B6F5D-C7F2-4E77-B701A9F646061CB3","STATUS":"Passed","PARENTID":"","SPECSTATS":[{"ERROR":{},"STARTTIME":1465879027043,"TOTALDURATION":501,"FAILORIGIN":{},"STATUS":"Passed","SUITEID":"4CBC4C98-BD18-4C0D-BE187715F5C1BFAB","ENDTIME":1465879027544,"NAME":"Tests list methods","ID":"0503EA96-88EA-4D5B-AF96773612180709","FAILMESSAGE":""},{"ERROR":{},"STARTTIME":1465879027544,"TOTALDURATION":392,"FAILORIGIN":{},"STATUS":"Passed","SUITEID":"4CBC4C98-BD18-4C0D-BE187715F5C1BFAB","ENDTIME":1465879027936,"NAME":"Tests the ability to retrieve the created user record","ID":"C96F286C-09F4-4082-A8083A594B0EA277","FAILMESSAGE":""},{"ERROR":{},"STARTTIME":1465879027936,"TOTALDURATION":461,"FAILORIGIN":{},"STATUS":"Passed","SUITEID":"4CBC4C98-BD18-4C0D-BE187715F5C1BFAB","ENDTIME":1465879028397,"NAME":"Tests the ability to update a user record","ID":"28DDD96E-24E5-4081-BF77E11444F8067F","FAILMESSAGE":""}],"ENDTIME":1465879028397,"TOTALERROR":0,"NAME":"Runs tests against created user","ID":"4CBC4C98-BD18-4C0D-BE187715F5C1BFAB","SUITESTATS":[]}],"GLOBALEXCEPTION":""}],"totalPass":4,"totalDuration":2455,"totalSkipped":0,"totalFail":0,"totalSpecs":4,"labels":[],"endTime":1465879028497,"totalError":0,"totalBundles":2}

★Running in Sublime Text 2

★Install PackageControl into Sublime Text

★Install Grunt from PackageControl

○https://packagecontrol.io/packages/Grunt

★Update Grunt Sublime Settings for paths

{

"exec_args": { "path": "/bin:/usr/bin:/usr/local/bin” }

}

★Then Ctrl / Command Shift P – grunt

https://packagecontrol.io/packages/Grunt

https://packagecontrol.io/packages/Grunt

★Running in Sublime Text 2

★Continuous Integration

Travis CI

Travis CI

Jenkins

Jenkins

★Q&A

★Any questions?

Technology

ITB2016 -BDD testing and automation from the trenches