Test automation @ bol.comA Prerequisite in a Fast Growing Development Organization

Joost van Wollingen & Elise van Krevelen

Agenda

• What was the problem?

• How did we fix it?

• What did it bring us?

Joost van Wollingen & Elise van Krevelen

• Test Engineer

• Back office applications

• Test Engineer & information analyst

• Front –end applications

@jpjwolli

> 25 million page visits per month

Our (hi)story

Back in 2012

WebshopCatalogueProduct Content

PricesAvailability

FullfillmentDeliveryReturns

Check for fraudStock management

etc..

The shop!Add to cartCheck outPayments

PromotionsGift cards

etc..

Fulfillment

Catalogue

Our suppliers

Service oriented architecture

The evolution of test automation

2012- 2013 2013 - 2015 2015 -

Selection process

Testers

Developers

Product Owners

“We need quick test runs with quick results”

“I’d like to use a language I’m familiar with”

“I want to do a quality check in between releases”

“I want to be notified when High priority tests are failing” Management

Operations“ A good reporting tool is very important to me”

And the winner is …Fitnesse & Selenium

• Open-source

• Flexible; adaptable

• Java-based

• Easy to read

Introducing a new tool

Ensuring adoption

Reliable tests

Technical & functional support

Gamification

Team involvement

TA support team

FixturesSolve common problemsStubs

Fitnesseslaves

Enable scrum teams

Reporting

Knowledge sharing Innovation

Mobile testing

Test reporting

Depth ofExpertise

Breadth of knowledge

Role of the testerShape of the ideal information worker

Role of the tester

• Boundaries between roles are fading

• Creating automated checks is not solely a testers job

• Tester stays in the ‘directors chair’

Results so farThe Good, The Bad & The Ugly

Results so far, The Good

Ratio test engineers vs. software engineersUser stories vs. scrum team members

Results so far, The Good

Incidents vs. ratio incidents per scrum teamAutomated test cases in 2014

Results so far, The BadNot quite

Results so far, The Ugly

• There is no single ‘right’ way to do it

• It can be hard to keep things DRY when starting out

• Check automation has to become part of the DoD

Reaching the limitsSingle fitnesse repository

Agile in test automation

2015 -

The evolution continues

Takeaways

• Allow yourself to make mistakes and dare to change course

• Involve as many different stakeholders as possible

• Trust in the expertise of your scrum teams. You will be rewarded!

Thanks!till next bol.com

Joost van Wollingen

Elise van Krevelen