Upload
sergio-arroyo
View
223
Download
0
Embed Size (px)
Citation preview
SOME TESTING
The miseries and great hardships of software testingSergio Arroyo Cuevas - @delr3ves
“I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence …
-Kent Beck
WHO AM I?
WHO AM I?
MOTIVATION
DISCLAIMER
No silver bullets
What is testing about?
Getting feedback,as often as we can
the sooner, the better
So please, send me your’s@delr3ves
WHAT ARE WE TALKING ABOUT?
TAXONOMY
Taxono
my
Scope
Unit
Integra3on
System
VisibillityBlackbox
Whitebox
Inten3on
Acceptance
Func3onal
NonFunc3onal
…
TechniqueSta3c
Dynamic
Execu3onAutoma3c
Manual
Google’s
Small
Medium
Big
TESTS BY SCOPE
SCOPE
Database
PORT1 PORT2
ADAPTER1
UNIT TEST
DEPENDENCY INJECTION
➤Dependency:A depends on B when A needs B to do its job.
➤Injection:Object (C) which uses A will tell A who is B.
DEPENDENCY INJECTION
➤Dependency:A depends on B when A needs B to do its job.
➤Injection: Object (C) which uses A will tell A who is B.
DEPENDENCY INJECTION
CA
B
➤A depends on B ➤C will tell A who is C
DEPENDENCY INJECTION
➤ Separate business logic from creation logic ➤Avoid use of new for service objects. ➤Value objects can be created any where.
➤ Service objects in charge to implement business logic.
➤ IOC Container or factories in charge of creation logic.
DEPENDENCY INJECTION
public UserService(UserValidator userValidator, UserDao userDao) { this.userValidator = userValidator; this.userDao = userDao; }
public User createUser(User user) throws ValidationException { this.userValidator.validate(user); user = this.userDao.create(user); return user; }
public User createUser(User user) throws ValidationException { UserValidator userValidator = new UserValidator(...); userValidator.validate(user); UserDao userDao = new UserDao(...); user = userDao.create(user); return user; }
VS
DEPENDENCY INJECTION
public UserService(UserValidator userValidator, UserDao userDao) { this.userValidator = userValidator; this.userDao = userDao; }
public User createUser(User user) throws ValidationException { this.userValidator.validate(user); user = this.userDao.create(user); return user; }
public User createUser(User user) throws ValidationException { UserValidator userValidator = new UserValidator(...); userValidator.validate(user); UserDao userDao = new UserDao(...); user = userDao.create(user); return user; }
VSthis sucks
TEST DOUBLES
TEST DOUBLES (FAKE)
Fake implementation in order to make test pass.
public UserDaoFake implements UserDao {
public User create(User user) { return ...; } }
TEST DOUBLES (STUB)
Stubs provide canned answers to calls made during the test, usually not responding at all to anything outside what’s programmed in for the test.
UserValidator validatorMock = mock(UserValidator.class); stub(validatorMock.validate(any(User.class))) .toThrow(new ValidationException());
var validateCall = Sinon.stub(); validatorStub.withArgs(user) .onFirstCall().returns(validationError);
var userValidator = { validate: validatorStub; }
OR WITH JS
TEST DOUBLES (SPY)
Spies are objects that also record some information based on how they were called
var validatorSpy = Sinon.spy(); var userValidator = { validate: validatorSpy; } userValidator.validate(user); sinon.assert.calledOnce(validatorSpy); sinon.assert.calledWith(validatorSpy, user);
OR WITH JS
UserValidator validatorSpy = spy(new UserValidator()); doThrow(new ValidationException())
.when(validatorSpy).validate();
TEST DOUBLES (SPY)
Spies are objects that also record some information based on how they were called
var validatorSpy = Sinon.spy(); var userValidator = { validate: validatorSpy; } userValidator.validate(user); sinon.assert.calledOnce(validatorSpy); sinon.assert.calledWith(validatorSpy, user);
OR WITH JS
UserValidator validatorSpy = spy(new UserValidator()); doThrow(new ValidationException())
.when(validatorSpy).validate();
TEST DOUBLES (MOCKS)
Informal: think in a Stub which is also a Spy.
It also responds with default values to non-explicitly declared methods
UserValidator validatorMock = mock(UserValidator.class); when(validatorMock.validate(any(User.class)))
.thenTrhow(new ValidationException()); verify(validatorMock).validate(any(User.class))
var validatorAPI = {validate: function()}; var validatorMock = Sinon.mock(validatorAPI); validatorMock.expects('validate').once() .withArgs(user).throws(validationError) validatorAPI.validate(user) validatorMock.verify()
OR WITH JS
INTEGRATION TEST WHICH WANT TO BE UNIT TESTS
FIRST(IT)
➤ Fast ➤Hundreds or thousands per second
➤ Isolates ➤Failure reasons become obvious
➤ Repeatable ➤In any order, any time
➤ Self-validating ➤No manual execution required
➤ Timely ➤Written before code
➤ Immutable*
➤SUT is in the same state after execute the tests
➤ Trusted* ➤When the test fails, the system fail and when the test works, the system works
INTEGRATION TEST WHICH WORKS WITH EXTERNAL SYSTEM
Database
Fast Isolates
Repeatable Self-Validating
Timely Inmutable*
Trusted*
INTEGRATION TEST WHICH WORKS WITH EXTERNAL SYSTEM
Database
Fast Isolates
Repeatable Self-Validating
Timely Inmutable*
Trusted*
INTEGRATION TEST WHICH USES THE UI
Database
Fast Isolates
Repeatable Self-Validating
Timely Inmutable*
Trusted*
INTEGRATION TEST WHICH USES THE UI
Database
Fast Isolates
Repeatable Self-Validating
Timely Inmutable*
Trusted*
SYSTEM TEST
Database
Fast Isolates
Repeatable Self-Validating
Timely Inmutable*
Trusted*
SYSTEM TEST
Database
Fast Isolates
Repeatable Self-Validating
Timely Inmutable*
Trusted*
WHO, WHEN AND WHERE RUN THE TESTS?
➤ Unit ➤ Owner: developer ➤ When: after every change ➤ Where: every computer
➤ Integration ➤ Owner: developer || QA team ➤ When: as part or after commit stage ➤ Where: devel and pre-pro environments
➤ System ➤ Owner: QA team ➤ When: after commit stage ➤ Where: devel and pre-pro environments
STRATEGIES
STATIC EVALUATION
➤Informal review ➤Formal review (inspection) ➤Checklists ➤Successive abstraction
➤Walkthrough
DYNAMIC EVALUATION
➤White box ➤Path Coverage ➤Statement Coverage ➤Condition Coverage ➤Function Coverage
➤Black box ➤Equivalence partitioning ➤Boundary values analysis
WHITE BOX (*-COVERAGE)
1. Get flow diagram of the SUT
2. Calculate cyclomatic complexity
3. Determine a data set which force going one path or another 4. Exercise the SUT with this dataset.
errors = []
if(user.name ==null||user.email == null) {
errors.push('mandatory fields not found');
}
//do the rest of whatever
for(var i=0; i < user.friends ; i++ ) {
errors.push(checkFriendShipt(user.friends[i]))
}
WHITE BOX (*-COVERAGE)
1. Get flow diagram of the SUT
2. Calculate cyclomatic complexity
3. Determine a data set which force going one path or another 4. Exercise the SUT with this dataset.
errors = []
if(user.name ==null||user.email == null) {
errors.push('mandatory fields not found');
}
//do the rest of whatever
for(var i=0; i < user.friends ; i++ ) {
errors.push(checkFriendShipt(user.friends[i]))
}
a
b c
d
…x
WHITE BOX (*-COVERAGE)
1. Get flow diagram of the SUT
2. Calculate cyclomatic complexity
3. Determine a data set which force going one path or another 4. Exercise the SUT with this dataset.
errors = []
if(user.name ==null||user.email == null) {
errors.push('mandatory fields not found');
}
//do the rest of whatever
for(var i=0; i < user.friends ; i++ ) {
errors.push(checkFriendShipt(user.friends[i]))
}
a
b c
d
…x
edges – nodes + 2 = predicate nodes +1 = number of regions = 4
BLACK BOX (PARTITIONING)
1. Identify equivalence classes 2. Select dataset:
1. Assign a unique value for every class
2. Select tests cases which cover the most valid classes
3. Select tests cases which cover only one invalid class at the same time
BLACK BOX (PARTITIONING)
1. Identify equivalence classes 2. Select dataset:
1. Assign a unique value for every class
2. Select tests cases which cover the most valid classes
3. Select tests cases which cover only one invalid class at the same time
BLACK BOX (PARTITIONING)
1. Identify equivalence classes 2. Select dataset:
1. Assign a unique value for every class
2. Select tests cases which cover the most valid classes
3. Select tests cases which cover only one invalid class at the same time
BLACK BOX (PARTITIONING)
Register
Username*
Password (6-10 chars including numbers)
BLACK BOX (PARTITIONING)
Username Password
U1: myNick P1: p4ssw0rdU2: “empty” P2: p4ss
P3: l4rg3p4ssw0rdP4: password
Register
Username*
Password (6-10 chars including numbers)
BLACK BOX (PARTITIONING)
Username Password
U1: myNick P1: p4ssw0rdU2: “empty” P2: p4ss
P3: l4rg3p4ssw0rdP4: password
Register
Username*
Password (6-10 chars including numbers)
Test Cases myNick, p4ssw0rd √ myNick, p4ss X myNick, l4rg3p4ssw0rd X myNick, password X “empty”, p4ssw0rd X myNick, p4sw0rd_* X
BLACK BOX (METHODS WHICH TELL US A STORY)
VS
BLACK BOX (METHODS WHICH TELL US A STORY)
but if you hate exceptions…
AUTOMATIC TESTING
4 PHASES-TESTS
1. Set Up 2. Exercise 3. Verify 4. TearDown
TESTING FRAMEWORKS FAMILIES
➤X-Unit ➤@Before(Class)
➤@Test
➤@After(Class)
➤Spec ➤describe
➤ beforeEach
➤ it
➤ afterEach
➤Specification by example (A.K.A BDD) ➤Given
➤When
➤Then
XUNIT@Before public void setUp() { this.userValidator = mock(UserValidator.class); this.userDao = mock(UserDao.class); this.userService = new UserService(userValidator, userDao); }
@Test public void createValidUserShouldNotFail() { //Exercise User expectedCreatedUser = new User("irrelevantUser"); when(userValidator.validate(any(User.class))); when(userValidator.validate(any(User.class))).thenReturn(createdUser); User createdUser = userService.create(new User()); //Assertions assertThat(createdUser, equalTo(expectedCreatedUser)); }
@Test(expected=ValidationException) public void createInvalidUserShouldFail() { when(userValidator.validate(any(User.class))) .thenReturn(new ValidationException()); userService.create(new User("irrelevantUser")); }
@After public void tearDown() { //clean the state here}
SPEC (SUITE PER CLASS)describe('UserService test suite:', function(){ beforeEach(function(){ // setup the SUT }) it('when create a valid user should not fail', function(){ // exercise + assertions }) it('when create an invalid user should fail', function(){ // exercise + assertions }) afterEach(function(){ // clean the state }) })
• UserService test suite:• When create a valid user should not fail √• When create an invalid user should fail √
The report will look like:
SPEC (SUITE PER SETUP)describe('UserService test suite:', function(){ describe("when create a valid user ", function() { beforeEach(function(){ // setup and exercise })
it('should return valid user', function(){ // partial assertions })
it('should call validator', function(){ // partial assertions })
it('should call dao', function(){ // partial assertions })
afterEach(function(){ // clean the state }) }) })
BDD (SPECIFICATION)
Feature: User registrationScenario: User tries to register sending valid data so the system will create new account Given the user has introduced <username> and <password> into the registration form And has accepted terms and agreements When send the registration from Then the user with <username> should be createdExample: | username | password | | myNick | p4ssw0rd |Scenario: User tries to register sending invalid data so the system will reject user Given the user has introduced <username> and <password> into the registration form And has accepted terms and agreements When send the registration from Then the system should notify <error>Example: | username | password | error | | myNick | p4ss | password should have at least 6 characters | | myNick | l4rg3p4ssword | password should have at less than 10 characters | | myNick | password | password should contains at least a number | | | p4ssword | username is mandatory |
BDD(IMPLEMENTATION)@given("the user has introduced (\w)+ and (\w)+ into the registration form") public void populateForm(String username, String password) { ... }
@given("has accepted terms and agreements") public void acceptTerms() { ... }
@when("send the registration from") public void sendRegistrationForm() { ... }
@then("the user with (\w)+ should be created") public void verifyUserIsCreated(String username) { ... }
@then("the system should notify <error>") public void verifyErrors(String error) { ... }
TESTABLE DESIGN
NON-TESTABLE DESIGN SMELLS (BY MISKO HEVERY*)
➤Constructor does Real Work
➤Mixing business and creation logic
➤Class does too much work
➤Digging into collaborations
➤Brittle Global State & Singletons
*See http://misko.hevery.com/attachments/Guide-Writing Testable Code.pdf
FROM NON-TESTABLE TO TESTABLE
➤First of all, go to http://refactoring.com/
➤I suggest: 1. Add integration regression test.
2. Remove new from methods and ad it to constructors (this will prepare your class for dependency injection).
3. Add a factory for Stateful services
4. Creates a constructor which receive every dependency your class will need.
5. Remove static classes and methods (adding the new non-static as a class dependency).
6. Add as much tests as you want to ;)
➤Important!!! Do it step by step
REMOVES NEW FOR DEPENDENCIES
public UserService() { this(new UserValidator(), new UserDao()); } public UserService(UserValidator userValidator, UserDao userDao) { this.userValidator = userValidator; this.userDao = userDao; }
public User createUser(User user) throws ValidationException { this.userValidator.validate(user); user = this.userDao.create(user); return user; }
public User createUser(User user) throws ValidationException { UserValidator userValidator = new UserValidator(...); userValidator.validate(user); UserDao userDao = new UserDao(...); user = userDao.create(user); return user; }
TO
ADD FACTORIES FOR STATEFUL DEPENDENCIES
public UserService() { this(new UserValidatorFactory(), new UserDao()); } public UserService(UserValidator userValidatorFactory, UserDao userDao) { this.userValidatorFactory = userValidatorFactory; this.userDao = userDao; }
public User createUser(User user) throws ValidationException { this.userValidatorFactory.getInstance(user).validate(); ... }
public User createUser(User user) throws ValidationException { UserValidator userValidator = new UserValidator(user); userValidator.validate(); ... }
TO
REMOVE STATIC CLASSES AND METHODS
class MyJsonSerializer { public static String serialize(Object myObject) { //put your magic here } }
class UserClient { public void createUser(User user) { Http.post(url, JsonSerizalizer.serialize(user)); } }
REMOVE STATIC CLASSES AND METHODS class MysonSerializer { public String serialize(Object myObject) { //put your magic here } }
class HttpClientWrapper { public void post(String url, String body) { Http.post(url, body); } }
class UserClient { private JsonSerializer serializer; private HttpClientWrapper httpClient
public UserClient() { serializer = new JsonSerializer(); httpClient = new HttpClientWrapper(); }
public void createUser(User user) { httpClient.post(url, serializer.serialize(user)); } }
TIPS & TRICKS
QUESTIONS & STUPID QUESTIONS
➤I spend more time writing code to setup my SUT than writing the test, how do you solve it?
➤I’ve never write a test, where can I start?
➤Should I verify every single interaction with my mocks?
➤Who tests the classes which test our classes?
➤Where I place my tests?
➤Could you be able to rewrite the code only reading the tests definitions?
➤What is the minimum coverage should I expect for my code?
I SPEND MORE TIME WRITING CODE TO SETUP MY SUT THAN WRITING THE TEST, HOW DO YOU SOLVE IT?
➤Read about Fixtures (Xunit Patterns is a good reference). ➤Fresh fixtures
➤Shared fixtures
➤Persistent fixtures ➤Think about new ways of organising your
tests.
I DUPLICATE TOO MUCH CODE ON OBJECTS CREATION, MOCKS DEFINITION AND ASSERTION…
➤Writing a lot of code to initialize value objects? ➤Create DataBuilders / MotherObjects
➤Writing a lot of code to initialize mock/stub objects? ➤Create Mock“Mother”
➤Writing a lot of asserts (TDD purist says only one assertion)? ➤Create CustomAsserts
User user = new User("irrelevantUsername", "v4l1dP4ss", [email protected]", ...);
VS User user = userMother.createValidUser();
assertNotNull(user.getUsername()); assertNotNull(user.getPassword()); assertNotNull(user.getEmail());
...VS
assertContainsAllMandatoryData(user);
I’VE NEVER WRITE A TEST, WHERE CAN I START?
I’ll bet you a beer that you called it *[Util|Helper]…
Database
PORT1 PORT2
ADAPTER1
SHOULD I VERIFY EVERY SINGLE INTERACTION WITH MY MOCKS?
It depends But if you feel you have to, it’s probably due
to a bad design :S
WHO TESTS THE CLASSES WHICH TEST OUR CLASSES?
➤Exactly, this is why it’s so important our tests follow
KISS
Q&APlace your question here!
@delr3ves
WHERE I PLACE MY TESTS?
➤Unit tests:
➤Test Class per Class
➤Test Class per SetUp (useful in Xunit frameworks)
➤Important naming convention (<ClassName>Test, <TestSuite>IntegrationTest, …)
➤System tests:
➤Different project
WHERE I PLACE MY TESTS?
Java Project (Test Class per Class) MyProject/ src/ main/ java/ com.groupId.artifactId.MyClass.java resources/ test/ java/ com.groupId.artifactId.MyClassTest.java com.groupId.artifactId.it.suite.MyTestCaseIntegrationTest.java resources/
NodeJs Project MyProject/ lib/ myClass.js main.js test/ ut/ /suite it/ lib/ myClassTest.js
Java Project (Class per SetUp) MyProject/ src/ main/ … test/ java/ com.groupId.artifactId.myclass.<SetUp1>Test.java com.groupId.artifactId.myclass.<SetUp2>Test.java …
WHERE I PLACE MY TESTS?
IOS ProjectMyIOSProject/ MyIOSProject/ ... app code ... MyIOSProjectTests/ ... test code ...
Android Project MyProject/ AndroidManifest.xml res/ ... (resources for main application) src/ main/
… (source code for main application) …test/
… (unit tests) …androidTest/
… (android specific tests / UI, etc) … tests/
AndroidManifest.xml res/
… (resources for tests) src/ ... (source code for tests)
¿COULD YOU BE ABLE TO REWRITE THE CODE ONLY READING THE TESTS DEFINITIONS?
➤Tests (specially Black Box tests) should tell us an story.
➤Use well defined, and complete scenarios for system tests:
➤Use business vocabulary for acceptance tests:
public void testValidaterUser1 { ... }
VS
public void validate_user_with_no_password_should_throws_error { ... }
com.mycompany.artifactId.it.TestSteps ...
VS
com.mycompany.artifactId.it.usermanagement.UserCreationSteps ...
WHAT IS THE MINIMUM CODE COVERAGE?
➤“… test as little as possible to reach a given level of confidence …”
➤Do not get obsess over test coverage, it’s a metric, not a goal.
It depends
Q&APlace your question here!
@delr3ves