Testing in a continuous delivery environment

Testing in a Continuous Delivery

EnvironmentSiddharth Somasundaram

Stefan Verhoeff

HERE

● Formerly known as Nokia / NAVTEQ Maps

● Powers Bing, Facebook and Yahoo Maps

● 4 out of every 5 cars with in-dash navigation are

powered by HERE

HERE

● Maps on the Web offering here.com

Testing Before Agile

● Testers are only involved in the later stage

● Long regression testing cycle before release

● Developers and Testers have separate work streams

○ Leads to conflicts between them

● Business pressure to release the software

○ Compromise on quality?

Importance of Agile

● Requirements change over time

● Testers are part of the planning process

● The entire team is responsible for Quality

● Faster release cycle

○ From once every 3 months to daily releases

Continuous Delivery

● Release early and often to production

● Automate testing and deployment

● Faster feedback, reduce risks and overhead for releasing

● Hot trend in the industry, especially web companies

But what about Quality Assurance?

● Lots of automation for regression testing

● Build enough trust in the test coverage

● Focus shifts from prevention of bugs to rapid discovery

and resolution

● Exploratory testing can be decoupled from releasing

How do we do it?● Team Culture and Technical practices we found to be useful

Team Culture Practices

● One agile Team, One Product and One Goal

○ Product Owner

○ Scrum Master

○ Developers / Testers

○ Offshore teams

● Communication solves most of the problems

Collaborate using Work Agreements

The team formulates rules and hold each other

accountable. For example:

● Running tests before committing

● No commits on red build

● First priority is to keep the build green

Build Monitor visible in every room

Pair Programming

Share knowledge with Tech Talks

Combine developer and operation teams

Technical Practices

Continuous Integration

Test Driven Development● Code and tests are developed at the same time

Build Pipeline

●

Test Coverage at all levels

© Alister Scott http://watirmelon.com/2011/06/10/yet-another-software-testing-pyramid/

Eliminate Flaky Tests

● Essential to keep the team

trusting the test results

● Mocking dependencies

Feature Flags

● No branching

○ Prevent integration problems

○ Quality built in

● Turn features on in production

when they are ready

● Quality boundaries for

different release stages

Logging & Monitoring

● What happens after we go live?

● Production monitoring ensures edge cases are covered

● Real-time metrics for: system, application, business

Logging & Monitoring

● Quickly responding to alerts

● Production is the ultimate proof the code works

Bug introduced We fixed it :-)

In Conclusion...

Is it Successful? The Metrics

2010 Now

Release quarterly Release daily

Mostly manual testing 7000+ unit tests, 400+ GUI tests

Preparing release takes weeks

Release package ready in 45 minutes

Deployment takes days Deploy 3 sites in 10 minutes

Management signoff required

Team empowered to make changes

Deployments 2012 vs 2013

Business Benefits

● Reduce risk involved in big changes

● Reduce time to market

● Smaller more frequent feature rollouts

● Quickly learn from feedback on product changes

Business Benefits

● Predictable releases

● Improved team morale

○ Less time spend in stressful firefighting

○ Less bug fixing, more feature delivery

○ Satisfaction of delivering to the customer

Challenges

● Everybody agreeing on the process

● Discipline comes over time only

● Feedback needs to be given quickly

● High investment in test automation and keeping it

running smoothly

● Need to react to incidents quickly

Future Plans

● Deploy every build to production

● Keep on investing in Automation and Monitoring

● Canary deployments to validate new releases before

global rollout

● Leverage quick cycle time for A/B testing driven feature

development

HERE ChecklistTeam Culture

● One Team

● Work Agreements

● Build Monitor in the

room

● Pair Programming

● Tech Talks

● DevOps

Technical

● Continuous Integration

● Test Driven Development

● Build Pipeline

● Test Coverage on all

levels

● Feature Flags

● Logging & Monitoring

Merci BeaucoupThank You

Danke SchönGrazie