Don't put your eggs in one basket - Lessons learnt from automation testing

Agile Testing, Test Automation & BDD, Singapore www.testingmind.com October 25, 2016

Don’t put your

Lessons learnt from automation testing

EGGS in onebasket

Brynley ScullyHead of Testing, South East AsiaHewlett Packard Enterprise

About the Speaker

2

Brynley Scully

Head of Testing, South East Asia

Hewlett Packard Enterprise

https://sg.linkedin.com/in/brynleyscully

@BrynleyScully

Project Testing Delivery

Testing Pre Sales

Testing Design & Review

Testing Solutioning

Project Remediation

Testing Governance

Test Management

Automation Test Management

Performance Test Management

Financial Information

Banking & Financial Services

Government

Agenda

Automation Testing Today

Five Lessons Learnt

Summary

3


4


Valuation of Global Automation Testing Market

5Source : Transparency Market Research

$ 12.9 B

Year 2015

$ 85.8 B

Year 2024

Asia Pacific will have largest growth rate

http://www.transparencymarketresearch.com/article/test-automation-market.htm


What skills do you need to be a good tester ?

6Source : 2016 State of Testing Report. Testing survey of more than 1000 participants in more than 60 countries.

of survey respondents feel that it is very important to have functional automation & scripting skills.61%


Does your organization have automation ?

7Source : 2016 State of Testing Report. Testing survey of more than 1000 participants in more than 60 countries.


Automation is key when transitioning to DevOps

8

The biggest change in testing as a result of the transition to DevOps, is Automation, as it is the core of the successful DevOps cycle.

VP of Applications

Logistics Provider, Italy.

“

”


Yet we have many challenges in automation testing:

Pitching the case for automation testing

What to automate

How much to automate

Re-use of automated test scripts

Maintaining automation test cases

9

Lesson #1:Don’t over-promise

10

Lesson #1Don’t over promise

11

Increased market share

Faster

Zero defects

Cheaper

100% automation

Fewer testers

Higher revenue

Automation testing promise basket

Magic

Eradicate world poverty


Why have Automation Testing in your projects ?

Understand the business objectives

Understand the technical landscape

12

We have business pressure to reduce our implementation cycles, for our retail banking app, from once every 2 months to once every 3 weeks.

Test Manager,

Banking, Thailand

“

”


Challenges faced in applying testing in agile development

13

29%

43%

0% 10% 20% 30% 40% 50% 60%

Lack of a good testing approach that fits withagile development method

Source : World Quality Report 2016-2017.

2016

2015


Build a realistic Automation Testing Approach/Strategy:

Scope - new automation test cases to build

Scope - existing automation test cases to maintain

Composition - manual vs automated test cases

Groupings of automated test cases

Frequency of automation test execution

Testing metrics to collect and publish

14


Be realistic in Automation Testing ROI

15

Effort

/ C

ost

Current – Manual Testing

Desired – Automation Testing

time



16

Effort

/ C

ost



time



17

Effort

/ C

ost



time

When will this happen ? How

much savings ?

How long will this take ?


Don’t put your eggs in one basket.

Don’t over-promise in automation testing, else you will have egg-static customers with unrealistic expectations.

18

Lesson #2:Don’t concentrate on automating UI test cases only

19

Lesson #2Don’t concentrate on automating UI test cases only

20

UI test case

UI test case

UI test case

UI test case

UI test case

UI test case

Automation test case basket


Challenges faced in applying testing in agile development

21

23%

41%

0% 10% 20% 30% 40% 50% 60%

Inability to apply test automation at differentlevels

2016

2015

Source : World Quality Report 2016-2017.


Mike Cohen

Automation Test Pyramid

22

Some

Lots

Fewer

API test cases

Unit tests

UI test cases


Mike Cohen

Automation Test Pyramid

Avoid the:

Inverted pyramid

Or

Ice Cream cone

23

Some

Lots

Fewer

API test cases

Unit tests

UI test cases

Some

Fewer

Lots

API test cases

Unit tests

UI test cases



Egg-span automation testing beyond just UI test cases.

24

Lesson #3:Don’t forget about automation frameworks

25

Lesson #3Don’t forget about automation frameworks

26

Automation test case






Automation test framework


27







Missing Automation test framework


28

Why:

Re-useable scripts

Easier script maintenance

What:

Data parameterization

Data separate from code

Log test results

Error Handling

Auto recovery/retry


29

Types of automation test frameworks:

Modular

Data Driven

Keyword Driven

Hybrid


weak

Don’t put your eggs in one ^ basket.

Egg-norring a good automation test framework will weaken your automation test re-use.

30

Lesson #4:Don’t just automate test cases only

31

Lesson #4Don’t just automate test cases only

32







Automation test basket


Automation testing is not just

the automation of test cases only

33


What else can we automate ?

34

Automate test reporting

Automate test data creation

Automate test environment provisioning


Emerging automation activities

35

Robotics Automation

Test Design Automation

Cognitive Automation

Machine learning

Predictive analysis



Uncover egg-stra opportunities for productivity through automation.

36

Lesson #5:Don’t lose faith in automation testing

37

Lesson #5Don’t lose faith in automation testing

38

Not many defects found

False sense of security

Many test execution failures

Not reliable

Takes time and effort to maintain

Automation Testing


39

Apply testing techniques:

Equivalence partitioning

Boundary Value Analysis

State Transition

Regenerate test data

Build randomness in scenarios

Complement with manual testing

Complement with exploratory testing

Use drivers and stubs



Egg-sercise complementary test techniques, data regeneration, randomness in your test scenarios, to avoid losing faith in automation testing.

40

Summary

41

Summary

42

1 Don’t over-promise in automation testing,

else you will have egg-static customers

with unrealistic expectations.

2 Egg-span automation testing beyond just

UI test cases.

3 Egg-noring a good automation test

framework will weaken your automation

test re-use.

4 Uncover egg-stra opportunities for

productivity through automation.

5 Egg-sercise complementary test

techniques, data regeneration,

randomness in your test scenarios, to

avoid losing faith in automation testing.

Technology

Don't put your eggs in one basket - Lessons learnt from automation testing