Test Automation and Test Automation and Keyword-driven testing

Brian Nielsen,

bnielsen@cs.aau.dk

Testing as a processg prequirements Planning&

Control

olog

i

specificationpreparation test plan

tekn

o

(abstract)Test

generation

abstract/logicaltest casesspecification

tions

t generation

Testimplemen-

tation

concrete/executableltest cases

orm

at

implementationTest

execution verdict/logs

Info

quality reportCompletion

CISS

Test Automationrequirements Planning&

Control

olog

i

specificationpreparation test plan

tekn

o

(abstract)Test

generation

abstract/logicaltest casesspecification

tions

t generation

Testimplemen-

tation

concrete/executableltest cases

orm

at

implementationTest

execution verdict/logs

Info

quality reportCompletion

CISS

1 Manual Testing1. Manual Testing+ easy & cheap to start+ flexible testing- expensive every execution

i i- no auto regression testing- ad-hoc coverage- no coverage measurement- no coverage measurement

M. Utting ”Practical Model Based Testing”

2 Capture Replay Testing2. Capture-Replay Testing+ flexible testing+ flexible testingexpensive first execution+ auto regression testing- fragile tests break easily- ad-hoc coverage- no coverage measurement

3 Script Based Testing3. Script-Based Testing+/- test impl. = programming+ automatic execution+ auto regression testing+ auto regression testingfragile tests break easily?

(depends on abstraction)(depends on abstraction)- ad-hoc coverage- no coverage measurement

4 Keyword Driven Testing4. Keyword-Driven Testing+ abstract tests+ automatic execution+ auto regression testing+ auto regression testing- robust tests- ad-hoc coveragead hoc coverage- no coverage measurement

5 Model-Based5. Model BasedTestingg

+ abstract tests+ automatic execution+ auto regression testing+ auto regression testing+ auto design of tests+ systematic coverage+ systematic coverage+ measure coverage of model and requirements- modelling overhead

Script based TestingScript-based Testing

Test inputs and observations (verdictassignment) programmed in scripts in g ) p g pdedicated or general-purpose languages.

+ Repeatable (for regression testing)+ Repeatable (for regression testing)+ Data driven testing: Fixed scripts, each parameterized

with different datad e e da a– Mixes test harness and logical test cases– Must be updated when specs (or impl) changeMust be updated when specs (or impl) change– Very lengthy– Good programmers may write well-structuredGood programmers may write well structured

reusable test code but ”it is just test code”

Keyword Driven Testingy g• Script based automation where test case

d i i t d f t tidesign is separated from automation– Focus on actions user/environment can do on

objects in SUT (at different abstraction levels)• ”Actions” appear in scripts,pp p• ”Action-code” implements the action

• aka “Action-word” testing (or table driven• aka Action-word testing (or, table driven testing), < 1994

• Black-box, subsystem, accept tests• Automated execution using a “framework”.Automated execution using a framework .

Keyword Driven Testingy g+ Concise, flexible, maintainable, + Read-/writeable by non-programmers

÷ Expressiveness of a scripting languageC t l t t d l t ti– Control structures and complex computations

– (Branching in test cases, complex data, matching of expected results)

Keyword Driven TestyObject Keyword Value

Eg. Sequence

LoginDialog Login ”badUserId”, ”GoodPass”

VerifyLogin ”Login failed”VerifyLogin Login failed

LoginDialog Login ”goodUserId”, ”badPass”

VerifyLogin ”Login failed”

LoginDialog Login ”goodUserId”, ”GoodPass”

VerifyLogin ”welcome”

W l Di l h L i ” U N ” ” d”WelcomeDialog changeLogin ”newUserName”,”newpwd”

confirm ”goodUserId”, ”newpwd”

confirmChange ”Change Failed”

KWD ProcessTest Expert Automation Expert

http://www.softwaretestinghelp.com/wp-content/qa/uploads/2010/01/keyword-driven-testing.pdf

Automation FixtureAutomation FixtureKeywod ScriptKeywod Script

Open(Scripts)

Engine/driverGeneric Libs

(fixtures)

While(!EOF){ Keyword, data=readFile();Case(keyword){L i id t > b it(d t )(fixtures) Login: widget->submit(data);

verifyLogin: …

Application Libs(keyword functions)

}Close()

SUT

}

Tools• SAFS, open source• EMOS Framework, open source• FIT/FitNesse open source• FIT/FitNesse, open source• Certify, Worksoft• Unified TestPro, SDT• TestFrame LogicaCMG• TestFrame, LogicaCMG• TestArchitect, LogiGear• Tools with built-in keyword support:

TestQuest Pro– TestQuest Pro,– QuickTest Professional

ExampleExamplehttp://localhost:8080/FitBookExamples.Chapter4TestingActions.Fig6TestChatServer

http://fit.c2.com/wiki.cgi?FitBookhttp://my.safaribooksonline.com/0321269349

ComparisonpCriteria Manual Script KWD Model-Based

Cost Development L M M LCost Development L M M L

Maintenance H? H L

Tools/equip L M M Hq p

Time Time to first test L M M M-H

Re-execution H L L L

Maintenance L H H L

Learn-ability

Skills L H M* H*ability Maturity L M M-H H

Readability L L M H*

I t t / Ch ll i L M M HInterest / Challenging L M M H

Effective-ness

Bug detection H L L H

Tracability L L M HTracability L L M H

Coverage L L M H

ReferencesReferences• http://en.wikipedia.org/wiki/Keyword-driven_testing• http://www.cs.waikato.ac.nz/~marku/mbt/• http://www.onestoptesting.com/automation-

framework/keyword-driven-testing/• http://www.softwaretestinghelp.com/wp-

content/qa/uploads/2010/01/keyword-driven-testing.pdf• http://www.musala.com/press/PR_2008_03_17/pr/Keyw

d d i %20T ti dford-driven%20Testing.pdf• Mark Fewster and Dorothy Graham, Software Test

Automation 1999 (Chapter 22)Automation, 1999 (Chapter 22) • M. Utting Practical Model-based testing