Testing in a DevOps World - Sogeti.nl in a DevOps World-Wolfgang Platz... · automation work, and by those seeking to support agile continuous automation practices.” “Integrated

© 2016 by© 2016 by .

Wolfgang Platz, Founder & CPO

Testing in a DevOps World

awards

Gartner MQLeader

© 2016 by

founded

2007

locations

AustriaGermanySwitzerlandUnited KingdomNetherlandsUSAAustraliaNew ZealandSingapore

support

TSIA 2014Staff Excellence

97% Renewal Rates

Forrester WaveStrong Performer

customers330+ large enterprise customers

calls us a Leader

“Tricentis delivers innovative tools that have provided strong results in driving automation success”.

� “Market-leading, model-based test planning and automation”

� “Market-leading test case design and planning with strong analytic and risk coverage model”

© 2016 by

“It should be considered by enterprises that have struggled to make test automation work, and by those seeking to support agile continuousautomation practices.”

� “Integrated synthetic test data management, service testing and support fortesting analytic applications”

The Future of Testing is …

© 2016 by

… but we can take it!

© 2016 by

Do the right things – and do things right.

Continuous Testing is a must.1

2

© 2016 by


Manual Testers become Automation Specialists.

2

3

The Digital Disruption …

© 2016 by

“Businesses must continuously exploit digital technologies to both create new sources of customer value and increase operational agility in service of customers.”

Forrester, March 2014, “The Future Of Business Is Digital”

… happens at a breathtaking pace

© 2016 by

Source: Tricentis customer, retail industry

Bi-modal business leads to a Bi-modal IT

Systems of

Innovation

Cha

nge

Governance

+ -

Mode 1

Mode 2

Agile

Gartner‘s Perspective:

© 2016 by

Systems of

Record

Cha

nge

Governance

- +

Mode 1

traditional

Agile

Systems of

Engagement

Cha

nge

+Mobile is on the

rise

Big Data and Cloud also

Technology

2x budget growth

over systems of record

BudgetEverybody talks about …

© 2016 by

Systems of

Record

Cha

nge

-Packaged is on

the rise

Sources: Forrester, The Midyear Global Tech Market Outlook For 2015 To 2016; Gartner, Forecast Alert: IT Spending, Worldwide, 2Q15

… you can’t make money without of enterprise IT budgets

80%

If the trend continues …

© 2016 by

Budget allocation of Systems of Record vs. Systems of Innovation; Enterprises, prognosis based on 3% IT growth and 3x growth of Innovation over Record

… unicorns will still be rare animals.

The importance of Systems of Record will not go away.

© 2016 by

Modern development and testing needs to fit all types of systems.

Agile

Waterfall

Others

Use of Software Development Methodologies, 2006 - 2013

?

?

© 2016 by

Agile

Agile development has taken over 2/3 of software development projects. Has it plateaued? Will it replace waterfall completely?

“Agile development for our Systems of Record and for quarterly releases? Really?

© 2016 by

Can’t they just stay waterfall?”

SVP QA, major US bank

How it all began …

© 2016 by

How it all began …

You remember the 80% Nightmare?

01/01/2015 31/05/2015myMajorTask80%From ToTask Name

© 2016 by

80% ready … and it took ages from there to get things done …

The law – 80:20 rule

… we implement 80%

© 2016 by

With 20% of effort …

implement 80% of functionality.

Split tasks!

01/01/2015 31/05/2015myMajorTask

100%

From ToTask Name

mySubTask1 01/01/2015 31/01/2015

© 2016 by

100%

Do not accept anything but 0% or 100%!

mySubTask1 01/01/2015 31/01/2015

mySubTask2 01/02/2015 28/02/2015 100%mySubTask3 01/03/2015 31/03/2015 0%…

Why agile development? – let’s stay waterfall!

In waterfall, customers see the value very late.

What if requirements change?

© 2016 by

t

50% of time/effort10% of value

The law – 80:20 rule

… we can provide 80%

of value.

© 2016 by

With 20% of effort …

of value.

Agile Development!

© 2016 by

t

+

In agile, value is delivered early.

50% of time/effort75% of value

Agile Development!

© 2016 by

t

Agility at different speeds

Systems of

Innovation

Cha

nge

Governance

+ -

Mode 1

Mode 2

Agile

Our perspective:

More

Agile

© 2016 by

Systems of

Record

Cha

nge

Governance

- +

Mode 1

traditional

Agile

Less

Agile

The Core Essentials of Agile

Ups

trea

mEpic/User Story

� Speed - Short iterations (sprints)� Communication - daily meetings (stand-ups)� Ownership - product owners� Feedback - early and frequent� Transparency - burn-down/burn-up

Core essentials

5

© 2016 by

Code/Config

� Transparency - burn-down/burn-up

Dow

nstr

eam

Delivery

� Continuous Integration (CI)

� Continuous Testing (CT)� Continuous Delivery (CD)

Core essentials

3

“Amazon deploys every 11.6 seconds.”

© 2016 by

“Amazon deploys every 11.6 seconds.”

Velocity 2011, May 2011: Jon Jenkins, "Velocity Culture"

Agility at different speedsStaging

area Production

Code/ Config

Epics/ User stories

© 2016 by

Release planning

Release Start

Fully tested release candidates hit the staging area

Deployment

GoLive

Upstream Downstream

Start EndSprint

How often will you deploy?Staging

area Production

Code/ Config

Epics/ User stories

?

© 2016 by

Release planning

Release Start

Fully tested release candidates hit the staging area

Deployment

GoLive

Upstream Downstream

Start EndSprint

?

The better metric (than deploys per day):

© 2016 by

Lead time = How long does it take from code committed to code successfully in production?

Agile Dev Operationsmeets

Dev Ops

© 2016 by

“DevOps is all about Continuous Testing!”http://devops.com/2015/02/02/devops-continuous-testing/

How to get there …


© 2016 by

How to get there …

What customers are asking1 …

“Are we testing too much?”

“Despite usage of so many automation tools, our cost and cycle time of testing

© 2016 by

is not reducing significantly. Why?”

1 … Source: Accenture customer survey, 2015

“We expect testing teams to provide useful insights and feedback whether we can go productive. What do numbers of test cases tell us?”

Customer Survey1

18% Everybody

© 2016 by

1 … Customer analysis Tricentis

18%average test automation

degree

Everybodywants to invest in test automation

© 2016 by

Automate tests.Do things right.

40%average risk coverage

achieved in comprehensive enterprise test portfolios1

67%average level of

redundancy in enterprise test portfolios1

© 2016 by

1 … Source: Customer projects, Tricentis

enterprise test portfolios1 test portfolios1

2 out of 3 test cases never yield defects.

60% of risk is secured through

hope & pray.

37.9%… only …

© 2016 by

Can you believe it?

37.9%… want to improve test

case quality

Do the mess for less?

© 2016 by

Automate tests.Do things right.

Do the right things. Have the right test cases.Test less.

© 2016 by

Do things right. Automate tests.Run faster.

Do the right things – Risk Based Testing

© 2016 by

Do the right things – Risk Based Testing

Today’s reporting is based on numbers of test cases.

Version 7.5 42,278 10,086 910

© 2016 by

What does it tell us?Is 53,274 the right number of test cases?

Can I go productive?

The law – (almost) 80:20 again

… we cover 80% of our

business risk.

© 2016 by

With 30% of test cases …

business risk.

66% 9% 15% 10%Core Bank

Risk Coverage [%]

© 2016 by

We need to introduce a new currency in testing.

Risk Coverage

Do things right - Automate

© 2016 by

Do things right - Automate

80 %Manual Tests Manual Testers

Manual testing is dominant.

© 2016 by 43

Automated UI Tests

API Tests

20 %Automated Tests

Automation Engineers

Today

Test automation requiresprogrammers.

Today, test automation is adevelopers discipline.

© 2016 by

So many tools you need to learn as a TA Developer …

© 2016 by

You sometimes feel like this?

© 2016 by

Remaining Manual Tests (Exploratory)

Automated UI Tests80 %

Manual Tests

< 15 %

Manual Tests

© 2016 by

UI

API

Today Target state(required)

API Tests

Orchestrated Service Virtualization

47

20 %Automated Tests

> 85 %

Test Automation

Set-up

Maintenance

Efficiency of Test Access

Task

4x

6x

FactorAPI

testing

25%

16%

UI test automation

100%

100%

© 2016 by

Runtime

Timing

6x

100+ x<1%

progressive

100%

regressive

Future Test Automation will focus on API testing.



Manual Tests

< 15 %

Manual Tests

© 2016 by

UI

API


API Tests


49

20 %Automated Tests

> 85 %

Test Automation

With the unprecedentedneed for automation,will today’s manual testersall become developers?

© 2016 by

all become developers?

Exploratory Tests Manual Testers

API Tests

Automated UI Tests

Automation Specialists(former Manual Testers) 10Some of them will.

How about all the

© 2016 by 51


API Tests

Automation Engineers 1

:

How about all the others?

Logical Name

Enter Vehicle Data

Make

Engine Performance [kW]

Number of Seats

Fuel

List Price [$]

Annual Mileage

Usage

Year of Construction

License Plate Number

Create Vehicle Insurance

Value

Audi

200

5

Petrol

35.000

10.000

Private

{Year}

B-CD 123

Action

Input

Input

Input

Input

Input

Input

Input

Input

Input

Model-BasedBrowser

Script-BasedBrowser

TestEasy to read. No programming skills.

TestEasy maintenance for UI & API tests.

TestMaximum reuse across technologies.

CodeSynchronization Behavior

CodeRecovery Handling

CodeData Driven Framework

1

2

3

© 2016 by

Enter Product Data

Next

Enter Insurant Data

Verify Quote Details

Next

Start Date

Insurance Sum [$]

Payment Option

Price List

Insurance Sum

Gross Premium [$]

{CLICK}

{CLICK}

01/03/2016

7Mio

Yearly

1.535,22

Input

Input

Input

Input

Input

Verify

Select

Select

Business Information Do you see the business information?

TestRobust against changes.

Maximum reuse across technologies. Data Driven Framework

CodeKeyword Driven Framework

3

4

Assert. AreEqual(" 1.535,22",driver.FindElement(By.XPath("//table[@id=' PriceList']/tbody/tr[ 10]/td[ 2]/b")).GetAttribute(" value"));

Logical Name

Enter Vehicle Data

Make


Number of Seats

Fuel

List Price [$]

Annual Mileage

Usage




Value

Audi

200

5

Petrol

35.000

10.000

Private

{Year}

B-CD 123

ActionAPI Test Case

Webservice

Logical Name

Enter Vehicle Data

Make


Number of Seats

Fuel

List Price [$]

Annual Mileage

Usage




Value

Audi

200

5

Petrol

35.000

10.000

Private

{Year}

B-CD 123

Action

Input

Input

Input

Input

Input

Input

Input

Input

Input

UI Test CaseBrowser

© 2016 by

Enter Product Data

Enter Insurant Data


Start Date

Insurance Sum [$]

Payment Option

Price List

Insurance Sum

Gross Premium [$]

01/03/2016

7Mio

Yearly

1.535,22

Business Information

Enter Product Data

Next

Enter Insurant Data


Next

Start Date

Insurance Sum [$]

Payment Option

Price List

Insurance Sum

Gross Premium [$]

{CLICK}

{CLICK}

01/03/2016

7Mio

Yearly

1.535,22

Input

Input

Input

Input

Input

Verify

Select

Select

Business Information

Manual Testers become Automation Specialists



Manual Tests

How much time have we got?

© 2016 by 54

UI

API


API Tests


20 %Automated Tests

3 years from now



Manual Tests

How much time have we got?

© 2016 by 55

UI

API

Today

API Tests


20 %Automated Tests

3 years from now

6 years from now

Tosca - process perspective

© 2016 by

© 2016 by

Tosca is a comprehensive and fully integrated suite.



2

© 2016 by


Manual Testers become Automation Specialists.

2

3

Thank You for your attention

Wolfgang PlatzStay in touch!

© 2016 by

Email: [email protected]

Follow us on Twitter: @tricentis

59

Documents

Testing in a DevOps World - Sogeti.nl in a DevOps World-Wolfgang Platz... · automation work, and by those seeking to support agile continuous automation practices.” “Integrated