Agile test automation

Bringing Agility in Test Automations

Hemant Kumar JaggiSenior Manager, MakeMyTrip.com

LET’S BE

AGILE !

Heard this before?

Thoughts on test automations the “Agile” way

To keep up the pace with agile application development

Traditional Test Automation approach isn’t working

Best practices must be incorporated everywhere

Ideally suited to an Ecommerce (or similar) environment

An Ecommerce looks like

Only thing that doesn’t change is “change”

Building Test Automations the “Agile” way

WHY?

BECAUSE TRADITIONAL

AUTOMATION APPROACH JUST DIDN’T

WORK FOR US

WHY? Challenges we face @ MakeMyTrip

Fast PaceReleases in a week?Monthly unique visitors?Monthly page views?

High Maintenance Cost & Test cases not always in “shape”

Challenges (contd.)

Complex Business RulesHotel promotion combinations?Cheapest Flights search?

Multiple GDS connections

Hotel Search engine?~10,500 Domestic Hotels~250,000 hotels across world

It’s impossible to reverse-engineer!

Challenges (contd.)

Execution issues:Automation execution may take 2-3

days to completeGolden data set takes ~7 days to

buildAutomation teams don’t know

domainFinally .. W

e don’t

trust

Autom

ation c

ode

Life of a QA Engineer is like

WHAT?

WHAT it should be?

Faster time to develop Automation

LOW maintenance costAgnostic to business logic“QA team” friendlyLesser complexityFaster execution time

“It just works ! ”

HOW?

HOW? In lines with Agile Manifesto..

Break into smaller storiesPrioritize them basis

Functional ImportanceFrequency of executionsDependencies

Work “with” QA teamDevelop a story and deliver

Create Buckets

Sanity testsEnd to end user flowsRegression tests*

Miscellaneous flows

Sanity Test

Breadth wise test coverage.MATS – Minimal Acceptance Test

SuiteRelease checklistDetermines whether build is

stable.

User Flows

Covers end to end user flowsDetailed breadth wise coverageE.g. different combinations of

booking flows.

But how to simplify regression tests?

Break them further ..

Into smaller flows, e.g. page wise test suites

Comparative TestingInteractive Testing

Comparative Testing

Comparative Testing

Works on two builds:“QA” build (i.e. build under test)“Production” build

Deploy them in same environment.

Read all ‘to-be-tested’ values from front end

Comparative Testing

Execute automation script on both builds

Dump them into XML (or excel)Excel template has formulae to

compare values from two buildsOr use an XML comparator (in

case of XML output format)

Comparative Testing

IF (Actual == Expected) THEN PASS

ELSE FAIL* Actual Value = value(QA build)* Expected Value = value(live build)

Take XML(or excel) as input to reporting framework, generate HTML reports

Let’s discuss

Pros & Cons of

Comparative Testing

Advantages of Comparative Testing

Faster turn around to developAutomation script does just what it is

supposed to doAutomation becomes a “dumb”

executionVery LOW maintenance costKeep it simple

Advantages of Comparative Testing

Agnostic to business layersChanges in business rules don’t

matter

High level of accuracyMakes it “thorough”Enhances confidence in QA team

Advantages of Comparative Testing

Overall progress is fasterMore dependency on Test Scenarios

than Test CasesLess/No domain knowledge required

for Automation team“Win-Win” for Automation and QA

team

Disadvantages of Comparative Testing

Any “live” issues won’t be caught

Execution requiresEither double the time (if sequential)Or double the resources (if parallel)

Interactive Testing

Interactive Testing

Complements comparative testing

Features that require user interaction

E.g. Filters, Sorting etc.

Comparative + Interactive equals “Regression”

Automation Coverage Tracking

Assign weights to each bucket, say

Automation Score = sum([(Tn/An)*100]*[Wn/(W1+W2+W3+W4+W5)]

Tn=Total Automated TC’s for a bucketAn=Total Automation feasible TC’s for a bucketWn=Weighted score of a bucket

Automation Coverage Tracking (Sample)

Computing efficiency gains (Sample)

Take Away

The “Agile” way of building Test Automations

We resolved “challenges” of traditional automation approach

Simplifying regressionRegression=Comparative + Interactive

Its working @ MakeMyTrip

Thank You

Hemant JaggiHemant.jaggi@makemytrip.com

Hemant.jaggi@gmail.com