Continuous Delivery Testing @HiQ

Continuous Delivery

Continuous Testing

Tomas Riha

Architect @ VGT/WirelessCar

Passionate about creativity, change and improvementHorrible at following instructions and performing repetitive tasks

MAJOR Project Liability

mail: triha74@gmail.comtwitter: @TomasRihaSE

blog: continuous-delivery-and-more.blogspot.com

Agenda

Why do we want to do Continuous DeliveryIntro to Continuous Delivery

Principles of Continuous DeliveryLook at a Pipe

Test Automation for Continuous DeliveryTest Driven Development and Continuous Delivery

Exploratory Testing and Continuous DeliverySome words on Tooling

Areas not covered

Why do we want to do Continuous Delivery

Our highest priority is to satisfy the customerthrough early and continuous delivery

of valuable software.

Why do we want to do Continuous Delivery

Decreased lead time not only improves our time to market but also our ability to learn and improve through a faster

feedback loop

Why do we want to do Continuous Delivery

As soon as we have added value to the application it needs to be in the hands of the customer or else we have just

added cost.

Why do we want to do Continuous Delivery

Reduced risk through delivery of small changes.

Are we really delivering as soon as possible?

Pre Planning

Dev Sys Test Reg Test

Pre Planning

Dev Sys Test Reg Test

Scrummerfall happens because its hard for developers to have something deployable for the testers to test.

System testing on something that has not been regression tested is fundamentally flawed.

Cost of regression test tends to bloat sprint content.

Sprint 2-4 weeks

Continuous Delivery

Build Release Deploy Test Prod

Your application always builds, passes its unit tests, deploys and passes system tests.

Continuous Integration/Delivery/Deploy

Continuous Integration Your application always builds and passes its unit tests.

Continuous DeliveryYour application always builds, passes its unit tests, deploys and passes its

system tests making it always ready to deploy to production through an automated process

Continuous DeploymentYour application automatically deploys to production through a fully

automated process

Continuously as in all the time?

Yes!

End of sprint isn't continuously. Weekly isn't continuously. Nightly isn't continuously.

Every code commit is continuously!

What Tests? Unit? Component? System?

Yes!

UnitComponent

SystemIntegrationRollback

LoadFailover

Principles of Continuous Delivery

1. The process MUST be repeatable, reliable and independent of key personnel

2. Automate everything

Principles of Continuous Delivery

3. If something is difficult or painful, do it more often

Principles of Continuous Delivery

4. Keep everything in source control AND release it

Principles of Continuous Delivery

5. Done means released

Principles of Continuous Delivery

6. Build Quality In

Principles of Continuous Delivery

7. Fail Fast

Principles of Continuous Delivery

8. Everybody has responsibility for the release process

Principles of Continuous Delivery

9. Improve continuously

Principles of Continuous Delivery

The pipe - Build once!

Build Release

Build and Unit test then release it!

The pipe - Release everything!

Build Release

Build Release

Build Release

Build Release

Code

DB Scripts

Server Config

Deploy script

Feedback Feedback

The pipe -Bundle released artifacts

Build Release

Build Release

Build Release

Build Release

Assembly

Feedback

The pipe - Use Same Deploy Mechanism

Function TestAssembly Deploy

DB

Server

Function Test

Function Test

Pipe Status

Feedback

Feedback Feedback

The pipe - Use Same Deploy Mechanism

Deploy

DB

Server

Pipe Status

Server

DB

Server Server

DB

Server Server

Deploy

Pre Prod Prod

UAT

FeedbackFeedback Feedback

Continuous Delivery

Build Release Assemble Deploy Deploy PreProd/ProdTest

Summary

We build onceRelease everything

Automate everythingFast feedback

Automate all tests except UAT

Obvious benefits

Continuous Regression Testing gives instant feedback.

Continuously deploying to test servers tests deploy mechanism several hundred times per release.

Always ready to push new release into UAT

We can get customer feedback as soon as value has been added

Automate all tests!

Wait a minute lets look at what we have been doing in the past.

Testing in the pastHow its been done

SYSTEM

GUI

Testing in the pastWhat we have done

100s of manual tests that..... verify our requirements

... ensures bad input doesn't break system... data is stored in the right place

... find bugs

Testing in the pastWe have automated it!

We have tried to automate the 100s of manual tests that..... verify our requirements

... ensures bad input doesn't break system... data is stored in the right place

... find bugs

We do it through instrumenting the GUI and verifying Database, GUI and System resources.

With very bad results....

Robust and Fast Test Automation

Tests that execute 100s of times per day have to be...

... executable without any human interaction... robust and give the same response every single time

... robust and have to survive refactoring... fast and parallelizable as adding 10 tests should not slow

down the feedback

We need to change the way we look at Test

On System Level we do verification of Use Case Acceptance Criterias. Requirement Verification.

On Component Level we do testing of Component Functionality. Testing.

We need to change the way we look at Architecture

Remember “Build quality in”.

We need to improve testability of our applications through...

...isolation of functionality into independent services with life cycles of their own.

...building interfaces for defined test points.

Building testable systems

Internal Service

GUI

Internal Service

Internal Service

Service API

Component Tests

Internal Service

Internal Service

Internal Service

Each Service Tested as Black Box in isolationand with high detail level

GUI is a component

GUI

Service API Mock Impl

GUI Test are much faster and robust if they test the GUI in isolation. High detail level.

Requirement Verification

Internal Service

GUI

Internal Service

Internal Service

Service API

Verifying Use Case acceptance criterias.

Low level of detail

Continuous Delivery

Build Release Deploy Test

Build Release Deploy Test

Build Release Deploy Test

Build Release Deploy Test

Build Release Deploy Test

Assemble Release Deploy Test

Component Pipe

100s of tests run in parallel testing

System Pipe

10s Use Case Requirements Verified

Test Driven Development

Pre Planning

Dev Sys Test Reg Test

Pre Planning

Dev Sys Test Reg Test

Scrummerfall happens because its hard for developers to have something deployable for the testers to test.

System testing on something that has not been regression tested is fundamentally flawed.

Cost of regression test tends to bloat sprint content.

Sprint 2-4 weeks

Test Driven Development

Pre Planning

Dev

Test Automation

Reg Test

Design Requirement Verification and Tests on multiple levels with the feature.

Develop test automation in parallel with feature development.

Cross functional pairing to improve quality of Tests, Test Architecture and Feature Testability

Pre Planning

Reg Test Reg Test

Verification

Verification

Test Driven Development

Who should automate the tests?

Test Design - Done by TestersTest Automation - Done by ????Coding - Done by Developers

Test Driven Development

Tests as Code

Test Automation IS Code and needs to be treated as first class application.

Test Architecture is super important and needs to be worked with in parallel with Application Testability

Architecture.

Test Driven Development

Tests as Code

Internal Service

DSL

Glue Code

SpecificationText or Code based Specification

Implementation of Specification

Domain Specific Language packaging service testability

Test Driven Development

Internal Service

DSL

Glue Code

SpecificationGive a valid user name and passwordwhen login is calledthen user is authorized

boolean authorized = loginService.authorizeUser(“triha”,”cdrockz”);assert(authorized, true);

public boolean authorizeUser(String userName, String password){ //Implementation of REST call to internal service return true; }

Test Driven Development

Tests as Code

Specification & Glue Code IS the Test

DSL is testability

Test Automation is Code

Test Driven Development

Pre Planning

Dev

Test Automation

Reg Test

Specifications of Requirement Verification written during Pre-Planning.Specification of Component Tests written during Pre-Planning.

Implementation of DSL and Glue Code in parallel with feature development.Implementation of Component Tests in parallel with feature development.

Pre Planning

Reg Test Reg Test

Verification

Verification

Test Driven Development

Tomorrow Agile Team Member

Developers with Testing skills Testers with Development skills

WILL BE SUPER HOT AND EARN SUPER NICE $$$$

Will be the only ones who get assignments

Exploratory Testing

Automate Everything!

Yes!

Everything that needs to be done in order to release a system into production.

Yet we still need Exploratory testing.

Exploratory Testing

Has been used to cover up manual regression testing

or worse

A way to get away with non reproducible manual regression testing

Exploratory Testing

Test Automation ensures we build the AGREED application

Exploratory Testing ensures we IMPROVE the application

Exploratory Testing

Exploratory Testing should be...

...done without knowledge of requirements....in time based sessions with specific focus area....outside the release lifecycle of the application.

Benefits

Continuous Regression testing gives instant feedback.

Continuously deploying to test servers tests deploy mechanism several hundred times per release.

Always ready to push new release into UAT

We can get customer feedback as soon as value has been added

Test Driven Development

Everyone is responsible for Quality

Sustainable Quality

Tools Used

We love to talk about what tools are best and spend ages arguing about it.

We need to understand what we must do before we pick the tools.

Still which ever tool we pick it will be insufficient and crap.

Tools Used

Don't use one tool to rule them all.

Cucumber and Fitnesse are ok for system level testing

RESTAssured, JUnit or pure Groovy, Ruby or Python scripts are often better for component level.

For Load testing one should use load testing tools such as JMeter

Continuous Delivery Process should execute tests through interface such as maven or gradle so that it can be Test Tool agnostic.

Areas Not Covered

To do Continuous Delivery well we also have to...

Reproducible Test EnvironmentsProduction like Test Environments

“Non functional requirements” as functional requirementsA/B testing

Power of Metricsand lots more...

Good Reading

Continuous Delivery by Jez Humble & Dave Farley

http://www.amazon.com/dp/0321601912?tag=contindelive-20

Thats it!

Feedback & Any questions you forgot to ask?

http://continuous-delivery-and-more.blogspot.se