Testautomatiseringserfarenheter

Presentation

Michael Albrecht Lars Wahlberg (Aguil)

Vad innehåller boken?

•  Michael Albrecht –  Anecdote 29.12 Cooperation Can Overcome

Resource Limitations •  Lars Wahlberg

–  Chapter 18 Automated Tests for Marketplace Systems: Ten years and Three Frameworks

Nu blir det vissa slides på Engelska !!!

Våra bidrag

•  Hur gick det till –  Hur vi blev kontaktade

•  Arbetet med att ta fram bok –  Massor med reviewer & uppdateringar –  Tekniska saker som att pillra med bilder

•  Legal process –  Skriva på kontrakt osv

•  Känslan att få boken (tillslut)

Framtagning av en bok

Första stegen

Anecdote 29.12 Cooperation Can Overcome Resource Limitations

•  Sit together •  Daily status meetings •  Learn some basic programming (same

language as the developers) •  Select tools already used by the developers

for unit testing if possible •  Create semi-automatic and automatic tests •  Learn from the past •  Get going!

Before the agile hype

•  55% a good tester •  35% development skills •  10% business or domain knowledge

Technical automater

•  http://www.addq.se/nyheter/back-to-the-future-an-agile-story-from-the-past/

•  http://www.automatedtestinginstitute.com

.and the rest of the story?

Chapter 18 Automated Tests for Marketplace Systems: Ten years and Three Frameworks

•  Market Place Systems •  Automated Tests •  Three frameworks •  ROI

Market Place Systems (MPS)

Gwy A

Gwy B

Gwy C

Gwy D

Matching Engine

Central Data

Query / History Server(s)

Trading Clients Protocol X

Trading Clients Protocol Y

Market Data

Admin Client

Primary Standby

Typical instruments: Equities, Commodities, Bonds, Derivatives …

Drivers for large nr of tests … •  Complex functionality •  Interference of functionality •  Critical functionality

–  Wrong auto match L –  Wrong transparency L –  Possible to “game” the system L

•  High uptime requirements •  Many customers (adaptations of a product)

Why automate?

•  Large nr of tests •  Development methods – many releases!

–  Incremental (10 years ago) –  Agile (now days)

•  Time to market important (rapid deliveries) –  Enhancements (new business opportunity) –  Patch (maintenance)

Solution => automate tests ...

Development of Test Framework

Abstraction Layer

Framework for automated tests

System

Test Scripts

Test Engine

Configuration

(ex JUnit)

Test Script

Enter Buy Order (25 SEK)

Enter Sell Order (24 SEK)

Verify Trade (25 SEK)

Enter Buy Order(price)

Enter Order (price, True, ...)

Enter Order (price, isBuy, …)

•  Create Enter Order Message

•  Fill in Price X

•  Fill in Buy or Sell

•  Fill in …

•  Send Transaction

Changes in the protocol can be handled in the abstraction layer, instead of rewriting ~100 of tests scripts …

Abstraction Layer – Example

Abstraction of Configuration

•  Predicates can be used Test Script

myList.add(new PredicateInstrument(“Equity”) )

myList.add(new PredicateCurrency(“USD”) )

myList.add(new PredicateTrader(myTrader1) )

myList.add(new PredicateTrader(myTrader2) )

OrderBook myOb = TestConfig.getAnyOb(myList)

Hint: Use TDD to design the Abstraction Layer !

If not found =>

Test Skipped !

Not same as Failed !

Development of Test Tool

•  Abstraction Layer is important –  Thin / Thick? Requirements on people skills! –  Who can modify? Who will maintain? –  When tests increased problems will be revealed!

•  Abstraction against test data –  Hard coded in a file, version controlled with tests –  Predicates may be used to increase robustness

Tip: Very good reference go to => www.stickyminds.com Search for “mexus” (author Lars-Ivar Sellberg)

Development of Tests Automatic Tests

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Week

Test

s

ActualPlanned

Test Execution (then)

Test Execution (then)

TestCase written before implementation

Now it works ! :)

Oops, it fails! :( Regression tests are good !

Oops, vacation!

Now it works again! :)

Framework - 1 •  Developed under much time pressure 1998 •  Tests written in Java

–  New language, attracted developers to test –  Thin Abstraction Layer

•  Own Test Engine (JUnit not available) •  Predicated used •  Simulations used (later more on that) •  After ~1 year: 2000 tests, ~3 year: 15 000 tests •  A real success !!!

Framework - 2 •  Developed 1999 based on success of -1 •  Development took some time …too much

–  Another system => new abstraction layer, still thin –  Predicates not used –  Switched to JUnit –  Focus on Tool instead of tests … few tests L

•  Heavy investment during 2002 –  Partly due to one senior Manager J –  Larger team => 50 000 tests after 3 years!

Framework - 3

•  Developed around 2003 •  Based on ideas from mostly from 1

–  But should be better! Func & Non Func tests, Preds –  Thick Layer – Tests in XML action language

•  Test Tool Architects became bottlenecks –  Difficult to do modifications of abstraction layer L

•  Re-done, Thin layer & No Predicates •  Now ~ 20 000 Tests

ROI - Savings Saving of Automization

-350

-300

-250

-200

-150

-100

-50

0

50

100

150

500

700

900

1100

1300

1500

1700

1900

nr of tests in batch

savi

ngs

in % Yearly

MonthlyWeeklyDaily

Effort dev manual test 2 h/testEffort dev automatic test 2 h/testTest Tool Maintanence 3000 h/yearEffort Execute manual test 0,1 h / testEffort Execute automatic test 8 h/batch

Utlottning av böcker !