Data-Driven and Keyword-Driven Test[1]

7/31/2019 Data-Driven and Keyword-Driven Test[1]

1/602

HELSINKIUNIVERSITYOFTECHNOLOGYDepartmentofComputerScienceandEngineeringSoftwareBusinessandEngineeringInstitute

PekkaLaukkanen

Data-DrivenandKeyword-Driven

TestAutomationFrameworks

Mastersthesissubmittedinpartialfulfillmentofthe

requirementsforthedegreeofMasterofScienceinTechnology.

Espoo,

February24,2006

Supervisor:ProfessorReijoSulonen


2/602

Instructor:HarriTohonen,M.Sc.


3/602

HELSINKIUNIVERSITYABSTRACTOFTHEOFTECHNOLOGYMASTERSTHESIS

Author:PekkaLaukkanenNameofthethesis:Data-DrivenandKeyword-DrivenTestAutomation

FrameworksDate:February24,2006Numberofpages:98+0Department:Department

ofComputerProfessorship:T-76ScienceandEngineeringSupervisor:Prof.ReijoSulonenInstructor:Harri

Tohonen,M.Sc.Thegrowingimportanceandstringentqualityrequirementsof


4/602

softwaresystemsareincreasingdemandforefficientsoftwaretesting.Hiringmoretestengineersorlengtheningthetestingtimearenotviablelong-termsolutions,rather

thereisaneedtodecreasetheamountofresourcesneeded.Oneattractive

solutiontothisproblemistestautomation,i.e.allocatingcertaintestingtasksto

computers.Therearecountlessapproachestotestautomation,andthey


5/602

workdifferentlyindifferentcontexts.Thismastersthesisfocusesononlyoneofthem,large-scaleframeworksforautomatedtestexecutionandreporting,butother

keyapproachesarealsobrieflyintroduced.Thethesisopensitsdiscussionoftest

automationframeworksbydefiningtheirhigh-levelrequirements.Themostimportantrequirementsareidentified

asease-of-use,maintainabilityand,ofcourse,theabilitytoautomatically


6/602

executetestsandreportresults.Moredetailedrequirementsarederivedfromthesehigh-levelrequirements:data-drivenandkeyword-driventestingtechniques,forexample,areessentialprerequisites

forbothease-of-useandmaintainability.Thenextstepinthethesisis

constructingandpresentingaframeworkconceptfulfillingthedefinedrequirements.Theconceptand

itsunderlyingrequirementsweretestedinapilotwherea


7/602

prototypeoftheframeworkandsomeautomatedtestsfordifferentsystemswereimplemented.Basedonthepilotresults,theoverallframeworkconceptwasfound

tobefeasible.Certainchangestotheframeworkandoriginalrequirementsarepresented,

however.Themostinterestingfindingisthatitispossibletocoverall

thedata-driventestingneedswiththekeyword-drivenapproachalone.Keywords:


8/602

testautomation,testautomationframework,data-driventesting,keyword-driventesting

ii


9/602

TEKNILLINENKORKEAKOULUDIPLOMITYONTIIVISTELMA

Tekija:PekkaLaukkanenTyonnimi:Aineisto-jaavainsanaohjatuttestiautomaatiojarjestelmatPaivamaara:24.2.2006Sivuja:98+

0Osasto:TietotekniikanosastoProfessuuri:T-76Tyonvalvoja:Prof.ReijoSulonenTyonohjaaja:

DIHarriTohonenOhjelmistojarjestelmienmerkityksensekalaatuvaatimustenkasvaminenaiheuttaapaineitaohjelmistojentestaukselle.Testaajien

maaranlisaaminentaitestausajanpi-dentamineneioleloputtomastimahdollista,


10/602

pikemminkinresurssejahalutaanvahentaa.Yksihoukuttelevaratkaisuontestauksenautomatisointieliosantestaustyonantaminentietokoneidenhoidettavaksi.Erilaisiatapojatestauksenautomatisointiinonlukuisiajane

toimivateritavoinerilaisissatilanteissajaymparistoissa.Tamadiplomityokasitteleetarkemminvainyhta

lahestymistapaa,laajojaautomaatiojarjestelmiatestienautomaattiseensuorittamiseenjaraportoimiseen,muttamyosmuuttavatovat

tarkeita.Testiautomaatiojarjestelmienkasittelyaloitetaanmaarittelemallaniillekorkeantasonvaatimukset.Tarkeimmiksi


11/602

vaatimuksiksitodetaanhelppokayttoisyys,yllapidettavyyssekatietenkinkykyautomaattisestisuorittaatestejajaraportoidaniidentulokset.Naidenvaatimustenkauttapaastaantarkempiinvaatimuksiinjatodetaanmm.etta

aineisto-ohjattu(data-driven)jaavainsanaohjattu(keyword-driven)testaustekniikkaovatedellytyksiasekahelppokayttoisyydelleettayllapidettavyydelle.Seuraavaksi

tyossasuunnitellaanmaaritellytvaatimuksettoteuttavatestiautomaatiojarjestelma.Jarjestelmantoimintaasekasenpohjanaolleita

vaatimuksiatestataanpilotissa,jossatoteutetaansekaprototyyppiitsejarjestelmastaetta


12/602

automatisoitujatestejaerilaisilleohjelmistoille.Pilotintuloksienperusteellasuunnitellunautomaatiojarjestelmanvoidaantodetaolevanpaaperiaatteiltaantoimiva.Lisaksikokemustenperusteellajarjestelmaansekaalkuperaisiinvaatimuksiinesitetaanjoitain

muutoksia.Mielenkiintoisinloydosonseettakaikkiaineisto-ohjatuttestitvoidaantoteuttaakayttaenainoastaan

avainsanaohjattualahestymistapaa.Avainsanat:testiautomaatio,testiautomaatiojarjestelma,aineisto-ohjattutestaus,avainsanaohjattutestaus

iii


13/602

Acknowledgements

ThismastersthesishasbeendoneforaFinnishsoftwaretestingconsultancycompanyQentinelmainlyduringtheyear2005.I

wishtothankmyinstructorHarriTohonen,M.Sc.andallotherQentineliansfor

comments,feedbackandpatience.

FromtheDepartmentofComputerScienceand

EngineeringIfirstofallwanttothankmyinstructor


14/602

ProfessorReijoSulonen.AdditionallyIamgratefulforJuhaItkonenandCasperLasseniusfortheirsupportandvaluablecomments.

Ialsowant

toexpressmygratitudetoMarkFewsterwhowaskindenoughtoreview

thethesisinitsearlyform.Markscommentsandpositivefeedbackmademe

believethattheideasIpresentarevalidandthe


15/602

remaininghardworkisworththeeffort.AlsoIwanttothankPetriHaapiowhohasbeeninchargeofboththeautomationproject

whereIgottheoriginalideaforthisthesisandanewone

wheretheautomationframeworkpresentedinthisthesishasbeensuccessfullytakeninto

realuse.

Finally,Iwouldliketothank


16/602

myfamilyandmywonderfulgirlfriendSonjaforeverything.Kiitos.

Espoo,February24,2006

PekkaLaukkanen

iv


17/602

Contents

Termsix

1Introduction1

1.1PromisesandProblemsofTestAutomation.........

..

.

.

.2

1.2DifferentTestAutomationApproaches...

..

.

.

.

.

.

.

.

.

.

.

.

.2

1.2.1Dynamicvs.StaticTesting...................5


18/602

1.2.2Functionalvs.Non-FunctionalTesting.............5

1.2.3GranularityoftheTestedSystem................6

1.2.4TestingActivities.........................8

1.2.5SmallScalevs.LargeScale

TestAutomation.........9

1.3Scope...................................11

1.4Methodology...............................


19/602

12

1.5Goals...................................12

1.6Structure.................................13

2RequirementsforTestAutomationFrameworks14

2.1HighLevelRequirements........................14

2.2FrameworkCapabilities.........................14

2.2.1

ExecutingTestsUnattended...................15

2.2.2StartingandStoppingTestExecution...

.

.

.

.

.

.

.

.

.

.


20/602

15

2.2.3HandlingErrors.........................15

2.2.4VerifyingTestResults......................16

2.2.5AssigningTestStatus......................16

2.2.6HandlingExpectedFailures...................

16

2.2.7DetailedLogging.........................17

v


21/602

2.2.8AutomaticReporting......................20

2.3Modularity................................21

2.3.1LinearTestScripts........................21

2.3.2TestLibrariesandDriverScripts...

..

.

.

.

.

.

.

.

.

.

.

.

22

2.3.3PromisesandProblems.....................23

2.4Data-DrivenTesting...........................23

2.4.1Introduction...........................23

2.4.2EditingandStoringTestData


22/602

.................24

2.4.3ProcessingTestData.......................26


2.5Keyword-DrivenTesting.........................27

2.5.1Introduction...........................

27

2.5.2EditingandStoringTestData.................27

2.5.3ProcessingTestData.......................29

2.5.4KeywordsinDifferentLevels..................29


2.6OtherImplementationIssues......................31

2.6.1ImplementationLanguage....................32


23/602

2.6.2ImplementationTechnique....................34

2.6.3TestwareArchitecture......................34

2.7Testability.................................36

2.7.1Control..............................36

2.7.2Visibility.............................38

2.8Roles....................................38

2.8.1TestAutomationManager....................

39

2.8.2TestAutomationArchitect...................39

2.8.3TestAutomator.........................40

2.8.4TestDesigner...........................40

2.9DetailedRequirements


24/602

..........................40

2.10ChapterSummary............................42

3ConceptforLargeScaleTestAutomationFrameworks43

3.1FrameworkStructure...........................43

vi


25/602

3.1.1TestDesignSystem.......................43

3.1.2TestMonitoringSystem.....................44

3.1.3TestExecutionSystem......................44

3.2PresentingandProcessingData-DrivenTest

Data..........47

3.2.1PresentingTestCases......................48

3.2.2UsingTestData.........................50

3.2.3Example..............................51

3.3PresentingandProcessingKeyword-Driven


26/602

TestData........55

3.3.1PresentingTestCases......................55

3.3.2PresentingUserKeywords....................57

3.3.3UsingTestData.........................58

3.4ChapterSummary............................61

4

ImplementationandPilot62

4.1ImplementationDecisions........................62

4.1.1

TechnicalDecisions........................62

4.1.2DecisionsRegardingthePilot..................63


27/602

4.2ImplementingReusableFrameworkComponents...........64

4.2.1TestDataParser.........................64

4.2.2Logger...............................66

4.2.3Summary.............................67

4.3

Data-DrivenWindowsApplicationTesting.........

.

.

.

.

.

.68

4.3.1


28/602

TestData.............................68

4.3.2DriverScript...........................68

4.3.3TestLibrary...........................71

4.3.4TestLog..............................71

4.3.5Summary

.............................71

4.4Keyword-DrivenWindowsApplicationTesting....

..

.

.

.

.

.

.72

4.4.1TestData

.............................72

4.4.2DriverScript...........................72


29/602

4.4.3TestLibrary...........................76

4.4.4TestLog..............................76

4.4.5Summary.............................76

vii


30/602

4.5Keyword-DrivenWebTesting......................79

4.5.1TestData.............................79

4.5.2DriverScript...........................79

4.5.3TestLibrary...........................81

4.5.4TestLog..............................81

4.5.5Summary.............................81

4.6

ChapterSummary............................85

5Results86

5.1FeasibilityoftheFrameworkConcept.

.

.

.

.

.

.

.

.

.

.


31/602

.

.

.

.

.

.86

5.2ChangestotheFrameworkandRequirements........

..

.

.

.87

5.2.1UsingOnlyKeyword-DrivenApproach

..

.

.

.

.

.

.

.

.

.

.

.

87

5.2.2SetUpandTearDown.....................89

5.2.3


32/602

TestSuites............................90

5.2.4GenericDriverScript......................90

5.3RevisedListofRequirements......................91

5.4ChapterSummary............................91

6

Conclusions93

Bibliography95

viii


33/602

Terms

AcceptanceTesting

ActionWordActualOutcome

AutomationAutomationFrameworkBaseKeyword

Black-BoxTesting

BugCaptureandReplay

Component

ComponentTesting

Aleveloftestingconductedfromtheviewpointofthecustomer,used

toestablishthecriteriaforacceptanceofasystem.Typically


34/602

basedupontherequirementsofthesystem.(CraigandJaskiel,2002)

Seekeyword.

Outputsanddatastatesthatthesystem

undertestproducesfromtestinputs.Seealsoexpectedoutcome.(FewsterandGraham,

1999)

Seetestautomation.

Seetestautomationframework.

Atermdefinedinthisthesisforkeywordsimplemented


35/602

inatestlibraryofakeyword-driventestautomationframework.Seealsouserkeyword.

Atypeoftestingwheretheinternalworkings

ofthesystemareunknownorignored.Testingtoseeifthesystem

doeswhatitissupposedtodo.(CraigandJaskiel,2002)

Seedefect.

Ascriptingapproachwhereatest


36/602

toolrecordstestinputasitissenttothesoftwareundertest.Theinputcasesstoredcanthenbeusedtoreproducethe

testatalatertime.Oftenalsocalledrecordandplayback.(BS7925-1)

Oneofthepartsthatmakeupasystem.Acollection

ofunitswithadefinedinterfacetowardsothercomponents.(IEEE


37/602

Std610.12-1990)

Testingofindividualcomponentsorgroupsofrelatedcomponents.(IEEEStd610.12-1990)

ix


38/602

Context-DrivenTestingAtestingmethodologythatunderlinestheimportanceofthecontextwheredifferenttestingpracticesareusedoverthepracticesthemselves.The

mainmessageisthattherearegoodpracticesinacontextbutthere

arenogeneralbestpractices.(Kaneretal.,2001)ControlScriptSeedriver

script.Data-DrivenTestingAscriptingtechniquethatstorestestinputs


39/602

andexpectedoutcomesasdata,normallyinatabularformat,sothatasingledriverscriptcanexecuteallofthedesignedtestcases.

(FewsterandGraham,1999)DefectIntroducedintosoftwareastheresultofan

error.Aflawinthesoftwarewithpotentialtocauseafailure.Also

calledfaultor,informally,bug.(CraigandJaskiel,2002;Burnstein,


40/602

2003)DomaincodePartoftheapplicationcodewhichcontainssystemfunctionality.Seealsopresentationcode.(Fowler,2001)DynamicTestingTheprocessofevaluating

asystemorcomponentbasedonitsbehaviorduringexecution.Seealsostatic

testing.(IEEEStd610.12-1990)DriverAsoftwaremodulethatinvokesand,perhaps,controls

andmonitorstheexecutionofoneormoreothersoftware


41/602

modules.(IEEEStd610.12-1990)DriverScriptAtestscriptthatdrivesthetestexecutionprocessusingtestingfunctionalityprovidedbytestlibrariesandmay

alsoreadtestdatafromexternalsources.CalledacontrolscriptbyFewster

andGraham(1999).ErrorAmistake,misconception,ormisunderstandingonthepartof

asoftwaredeveloper.(Burnstein,2003)ExpectedFailureOccurswhena


42/602

testcasewhichhasfailedpreviouslyfailsagainsimilarly.DerivedfromFewsterandGraham(1999).ExpectedOutcomeOutputsanddatastatesthatshouldresult

fromexecutingatest.Seealsoactualoutcome.(FewsterandGraham,1999)Failure

Inabilityofasoftwaresystemorcomponenttoperformitsrequiredfunctionwithin

specifiedperformancecriteria.Themanifestationofadefect.(IEEEStd


43/602

610.12-1990;CraigandJaskiel,2002)

x


44/602

FaultSeedefect.FunctionalTestingTestingconductedtoevaluatethecomplianceofasystemorcomponentwithspecifiedfunctionalrequirements.(IEEEStd610.12-1990)

FeatureAsoftwarecharacteristicspecifiedorimpliedbyrequirementsdocumentation.(IEEEStd610.12-1990)

FrameworkAnabstractdesignwhichcanbeextendedbyaddingmoreorbetter

componentstoit.Animportantcharacteristicofaframeworkthat


45/602

differentiatesitfromlibrariesisthatthemethodsdefinedbytheusertotailortheframeworkarecalledfromwithintheframeworkitself.The

frameworkoftenplaystheroleofthemainprogramincoordinatingandsequencing

applicationactivity.(JohnsonandFoote,1988)IntegrationTestingAleveloftestundertaken

tovalidatetheinterfacebetweeninternalcomponentsofasystem.


46/602

Typicallybaseduponthesystemarchitecture.(CraigandJaskiel,2002)KeywordAdirectivethatrepresentsasingleactioninkeyword-driventesting.Calledactionswords

byBuwaldaetal.(2002).Keyword-DrivenTestingAtestautomationapproachwheretest

dataandalsokeywordsinstructinghowtousethedataarereadfrom

anexternaldatasource.Whentestcasesareexecutedkeywords


47/602

areinterpretedbyatestlibrarywhichiscalledbyatestautomationframework.Seealsodata-driventesting.(FewsterandGraham,1999;Kaneret

al.,2001;Buwaldaetal.,2002;MosleyandPosey,2002)LibraryAcontrolled

collectionofsoftwareandrelateddocumentationdesignedtoaidinsoftwaredevelopment,use,

ormaintenance.Seealsoframework.(IEEEStd610.12-1990)Non-FunctionalTesting


48/602

Testingofthoserequirementsthatdonotrelatetofunctionality.Forexampleperformanceandusability.(BS7925-1)ManualTestingManuallyconductedsoftwaretesting.See

alsotestautomation.OracleAdocumentorpieceofsoftwarethatallowstest

engineersorautomatedteststodeterminewhetheratesthasbeenpassedor

not.(Burnstein,2003)PreconditionEnvironmentalandstateconditionswhichmust


49/602

befulfilledbeforeatestcasecanbeexecuted.(BS7925-1)

xi


50/602

PredictedOutcomeSeeexpectedoutcome.PresentationCodePartoftheapplicationcodewhichmakesuptheuserinterfaceofthesystem.Seealso

domaincode.(Fowler,2001)Quality(1)Thedegreetowhichasystem,component,

orprocessmeetsspecifiedrequirements.(2)Thedegreetowhichasystem,component,

orprocessmeetscustomeroruserneedsorexpectations.(IEEE


51/602

Std610.12-1990)RecordandPlaybackSeecaptureandreplay.RegressionTestingRetestingpreviouslytestedfeaturestoensurethatachangeoradefectfix

hasnotaffectedthem.(CraigandJaskiel,2002)RequirementAconditionorcapability

thatmustbemetorpossessedbyasystemorsystemcomponentto

satisfyacontract,standard,specification,orotherformallyimposeddocuments.


52/602

Canbeeitherfunctionalornon-functional.(IEEEStd610.12-1990)SetUpCodethatisexecutedbeforeeachautomatedtestcaseinoneparticulartest

suite.Arelatedtermusedinmanualtestingisprecondition.Seealsotear

down.SmokeTestingAtestruntodemonstratethatthebasicfunctionalityof

asystemexistsandthatacertainlevelofstability


53/602

hasbeenachieved.(CraigandJaskiel,2002)SoftwareTestAutomationSeetestautomation.SoftwareTestingTheprocessofoperatingasystemorcomponentunder

specifiedconditions,observingorrecordingtheresults,andmakinganevaluationofsome

aspectofthesystemorcomponent.(IEEEStd610.12-1990)StaticTestingTheprocess

ofevaluatingasystemorcomponentbasedonitsform,


54/602

structure,content,ordocumentation.Seealsodynamictesting.(IEEEStd610.12-1990)SystemTestingAcomprehensivetestundertakentovalidateanentiresystemandits

characteristics.Typicallybasedupontherequirementsanddesignofthesystem.(Craigand

Jaskiel,2002)SystemUnderTest(SUT)Theentiresystemorproducttobe

tested.(CraigandJaskiel,2002)

xii


55/602


56/602

TearDownCodethatisexecutedaftereachautomatedtestcaseinoneparticulartestsuite.Testautomationframeworksrunthemregardlessthe

teststatussoactionsthatmustalwaysbedone(e.g.releasingresources)should

bedonethere.Seealsosetup.TestAutomationTheuseofsoftware

tocontroltheexecutionoftests,thecomparisonofactual


57/602

outcomestopredictedoutcomes,thesettingupoftestpreconditions,andothertestcontrolandtestreportingfunctions.(BS7925-1)TestAutomationAframework

usedfortestautomation.Providessomecorefunc-Frameworktionality(e.g.loggingand

reporting)andallowsitstestingcapabilitiestobeextendedbyaddingnewtest

libraries.TestCaseAsetofinputs,executionpreconditionsand


58/602

expectedoutcomesdevelopedforaparticularobjective,suchastoexerciseaparticularprogrampathortoverifycompliancewithaspecificrequirement.(BS

7925-1)TestOracleSeeoracle.TestOutcomeSeeactualoutcome.TestRunnerA

genericdriverscriptcapabletoexecutedifferentkindsoftestcasesandnot

onlyvariationswithslightlydifferenttestdata.TestSuiteA


59/602

collectionofoneormoretestcasesforthesoftwareundertest.(BS7925-1)Test-DrivenDevelopment(TDD)Developmenttechniquewhereautomatedunittestsare

writtenbeforethesystemcode.Testsdrivethedesignanddevelopmentofthe

systemandacomprehensiveregressiontestsuiteisgotasaby-product.(Beck,

2003)TestabilityAcharacteristicofsystemundertestthatdefines


60/602

howeasilyitcanbetested.Consistsofvisibilityandcontrol.(Pettichord,2002)TestingSeesoftwaretesting.TestwareTheartifactsrequiredtoplan,design

andexecutetestcases,suchasdocumentation,scripts,inputs,expectedoutcomes,setup

andteardownprocedures,files,databases,environmentsandanyadditionalsoftwareorutilities

usedintesting.(FewsterandGraham,1999)

xiii


61/602


62/602

UnitApieceofcodethatperformsafunction,typicallywrittenbyasingleprogrammer.(CraigandJaskiel,2002)UnitTestingAlevel

oftestundertakentovalidateasingleunitofcode.Unittestsare

typicallyautomatedandwrittenbytheprogrammerwhohaswrittenthecodeunder

test.(CraigandJaskiel,2002)UserKeywordAtermdefined


63/602

inthisthesisforkeywordsconstructedfrombasekeywordsandotheruserkeywordsinatestdesignsystem.Userkeywordscanbecreatedeasily

evenwithoutprogrammingskills.WhiteBoxTestingTestingbaseduponknowledgeofthe

internalstructureofthesystem.Testingnotonlywhatthesystemdoes,but

alsohowitdoesit.(CraigandJaskiel,2002)xUnit


64/602

FrameworksFrameworksthateasewritingandexecutingautomatedunittests,providesetupandteardownfunctionalitiesforthemandallowconstructingtestsuites.

ThemostfamousxUnitframeworkisJUnitforJavabutimplementationsexistfor

mostprogramminglanguages.(Hamill,2004)

xiv


65/602

Chapter1Introduction

Softwaresystemsaregettingmoreandmoreimportantfororganizationsandindividualsalikeandatthesametime

theyaregrowingbiggerandmorecomplex.Itisthusonlylogicalthat

importanceofsoftwarequality1isalsorising.Softwarefaultshavecausedlossof

hugesumsofmoneyandevenhumanlives.Ifquality


66/602

doesnotgetbetterassystemsgrowinsize,complexityandimportance,theselossesareonlygettingbigger.(Burnstein,2003)

Theneed

forbetterqualitymeansmorepressureforsoftwaretestingandfortestengineers

takingcareofit.Testautomation,i.e.givingsometestingtaskstocomputers,

isanobviouswaytoeasetheirworkload.Computersare


67/602

relativelycheap,theyarefasterthanhumans,theydonotgettiredorbored,andtheyworkoverweekendswithoutextrapay.Theyare

notidealworkhorses,however,astheyonlyfinddefectsfromplaceswherethey

areexplicitlytoldtosearchthemandtheyeasilygetlostifsomething

inthesystemundertest(SUT)changes.Givingcomputersall


68/602

theneededdetailsisnoteasyandtakestime.(FewsterandGraham,1999)

Testautomationcanbeusedinmultipleways.It

canandshouldbeuseddifferentlyindifferentcontextsandnosingleautomation

approachworkseverywhere.Testautomationisnosilverbulleteitherbutithas

alotofpotentialandwhendonewellitcan


69/602

significantlyhelptestengineerstogettheirworkdone.(FewsterandGraham,1999)

Thisthesisconcentratesonlargertestautomationframeworksdesigned

fortestexecutionandreporting.Beforethescopecanbedefinedinmore

detailedmannersome

1Newtermsareemphasizedwhenusedforthe

firsttimeandtheirexplanationscanbefoundfromthe


70/602

listoftermsonpagesixxiv.

1


71/602

CHAPTER1.INTRODUCTION

backgroundinformationaboutdifferentautomationapproachesisneeded,however,andthatispresentedinSection1.2.Evenbefore

thatitistimetoinvestigateabitmorethoroughlywhytestautomation

isneededandwhatarethemainchallengesintakingitintouse.

1.1PromisesandProblemsofTestAutomationA


72/602

comprehensivelistoftestautomationpromises,aspresentedbyFewsterandGraham(1999),isshowninTable1.1.Similarpromiseshavebeenreportedalso

byotherauthorslikePettichord(1999),Nagle(2000)andKaneretal.(2001).

MostofthebenefitsinTable1.1canbesummarizedwith

wordsefficiencyandreuse.Testautomationisexpectedtohelp


73/602

runlotsoftestcasesconsistentlyagainandagainondifferentversionsofthesystemundertest.Automationcanalsoeasetestengineersworkload

andreleasethemfromrepeatingtasks.Allthishasthepotentialtoincrease

softwarequalityandshortentestingtime.

Allthesepromisesmaketest

automationlookreallyattractivebutachievingtheminreallife


74/602

requiresplentyofhardwork.Ifautomationisnotdonewellitwillbeabandonedandpromiseswillneverberealized.Alistof

commontestautomationproblems,againbyFewsterandGraham(1999),canbefound

fromTable1.2.

Thegeneralproblemwithtestautomationseemsto

beforgettingthatanylargertestautomationprojectisa


75/602

softwareprojectonitsownright.Softwareprojectsfailiftheydonotfollowprocessesandarenotmanagedadequately,andtestautomationprojects

arenotdifferent.Ofallpeople,testengineersoughttorealizehowimportant

itistohaveadisciplinedapproachtosoftwaredevelopment.(Kaner,1997;Zambelich,

1998;FewsterandGraham,1999;Pettichord,1999;Kaneretal.,


76/602

2001;Zallar,2001;Rice,2003)

1.2DifferentTestAutomationApproachesThissectionbrieflyintroducesmaintestautomationcategoriesasanintroductionand

backgroundfortherestofthisthesis.Thefocusedscopeandtargetfor

thisthesisaredefinedinnextsection.


77/602


RunexistingregressiontestsonanewversionofaprogramBeingabletorunpreviouslycreatedtests

withoutextraeffortclearlymakestestingmoreefficient.Runmoretestsmoreoften

Automationmeansfastertestexecutionwhichmeansmoretestrounds.Automationshouldalso

makecreatingnewtestcaseseasyandfast.Performtests


78/602

whichwouldbedifficultorimpossibletodomanuallyForexampleperformanceandstresstestsarenearlyimpossibletoconductwithoutautomation.Betteruse

ofresourcesAutomatingrepeatingandboringtasksreleasestestengineersformoredemanding

andrewardingwork.ConsistencyandrepeatabilityoftestsTestsarealwaysrunthe

samewaysotestresultscanbeconsistentlycomparedto


79/602

previousresultsfromprevioustestingrounds.Testscanalsobeeasilyrepeatedondifferentenvironments.ReuseoftestsReusingtestsfromearlierprojectsgives

starttoanewproject.akickEarliertimetomarketReusingtests

andshorteningtestexecutiontimefastensfeedbackcycletodevelopers.Intheend

thatshortensthetimetomarket.IncreasedconfidenceRunningan


80/602

extensivesetoftestsoften,consistentlyandondifferentenvironmentssuccessfullyincreasestheconfidencethattheproductreallyisreadytobereleased.

Table1.1:Commontestautomationpromises(FewsterandGraham,1999)


81/602


UnrealisticexpectationsManagersmaybelievetestautomationwillsolvealltheirtestingproblemsandmagicallymakethesoftware

qualitybetter.Automationexpertsshouldhelpmanagerssettingtheirexpectationsright.

PoortestingpracticeIftestingpracticesandprocessesareinadequateitisbetter

tostartimprovingthemthanbringingintestautomation.Automating


82/602

chaosjustgivesfasterchaos.

ExpectationthatautomatedAfterautomatedtesthasbeenrunsuccessfullytestswillfindalotofnew

onceitisnotverylikelytofindnewbugsunlessdefectsthe

testedfunctionalitychanges.Automatorsnor

mallyfindmoredefectswhilethey

aredevelopingteststhanwhentestsarere-executed.


83/602

FalsesenseofsecurityJustseeingatestreportwithnofailuresdoesnotmeanthattheSUTdidnothaveany.Testsmay

beincomplete,eithernottestingallfeaturesornotabletoseefailures

whentheyoccur.Testsmayalsohavedefectsandshowwrongresults.

MaintenanceWhentheSUTchangesalsoitstestschange.


84/602

Humantestengineersareabletohandleevenmajorchangeswithoutproblemsbutautomatedtestscanfailafteraslightestchange.Ifmaintainingtest

automationsystemtakesmoretimethantestingmanuallyitwillsurelybeabandoned.

Thesamewillhappenalsoifaddingnewfeaturestotheautomationsystem

istoocumbersome.

TechnicalproblemsBuildingandtaking


85/602


86/602

inmanyprocesses.

Table1.2:Commontestautomationproblems(FewsterandGraham,1999)


87/602


88/602

documentandcodereviewsandstaticcodeanalysis

(e.g.syntaxcorrectness,codecomplexity).Reviewsarebytheirnaturedonemainlybyhumans

whilestaticanalysisismostlyleftforcomputersandthereareplentyof

goodtoolsavailable.(IEEEStd610.12-1990;Burnstein,2003)Ibelievestatictestingis

importantandcangreatlyhelpinfindingdefectsevenin


89/602

earlystagesofprojects.Inthisthesis,however,interestisonlyondynamictesting.

1.2.2Functionalvs.Non-FunctionalTestingDynamictestingcan

befurtherdividedintofunctionalandnon-functionaltesting.Aimoftheformer,as

thenameimplies,isensuringthatthefunctionalityofthesystemadheresto

requirements.Thelatterisconductedtoverifythatother,non-functional,


90/602

aspectsofthetestedsystemworkaswell.(IEEEStd610.12-1990;Burnstein,2003)

Generallyallfunctionaltestingisdoneintwosteps

asillustratedFigure1.1.Firstsomethingisdonetothetestedsystem.Next

thetestoutcome,outputsproducedbythesystemandchangesinsystemsstate,

isverifiedagainstpredefinedexpectedoutcome.Inmostcasesboth


91/602


92/602


liabilityandmemorymanagement.Thesetestingareasarereallyimportantandfailuresinanyofthemmayruin

otherwisegoodandfunctionallycorrectproduct.Non-functionaltestingdiffersquitealotfrom

functionaltestinganditisalsoratherheterogeneousitself.Someareas(e.g.usability)

aremainlytestedmanuallybyexpertswhileothers(e.g.performance)


93/602

arenearlyimpossiblewithoutadequateautomationtools.

Inthisthesisthescopeisonfunctionaltestingbutmanyideaspresentedarerelevant

alsowhenautomatingnon-functionaltesting.

1.2.3GranularityoftheTestedSystem

Traditionallytestingisdividedintodifferentlevelssuchasunit,integration,systemand

acceptancetesting(Dustinetal.,1999;CraigandJaskiel,2002;


94/602

Burnstein,2003).Inmyopinionthisdivisionisnotverypracticalintestautomationcontext,however,becausesameautomationapproachcanbeusedin

manylevels.Forexamplesystemandacceptancetestingdonothavemuchdifference

fromthisperspective.ThatiswhyIpreferclassificationbasedonthegranularity

ofthetestedsystemintounit,componentandsystemtesting,


95/602

assuggestedbyMeszaros(2003).

UnitTesting

Thesmallestbuildingblockofanysystemisaunit.Unitshavean

applicationprogramminginterface(API)whichisusedwheninteractingwithotherunitsand

canalsobeusedtotestthem.Unittestingisinmostcases

besthandledbydeveloperswhoknowthecodeundertest


96/602


97/602

unitinterfacescanbedrivenwithoutanyspecialtools.Inthislevelinterfacesalsotendtobemorestablethaninhigherlevels,especially

inuserinterfacelevel,andchangesinotherunitshavelittleorno

effectunlessunitsarecloselyconnected.Findingandfixingdefectsisalsocheapest

inunitlevel.

Defactotesttoolsin


98/602

unitlevelaremultiplexUnitframeworks.ThefirstofthemanywasSmalltalkUnit(Beck,1994)whichwassoonportedtoJavaasJUnitby

ErichGamma(BeckandGamma,1998).AfterJUnitsbigsuccessimplementations


99/602


forotherlanguagessuchasPyUnitforPythonandNUnitforMicrosofts.NETfollowedshortly.(Hamill,2004)

Unittestingcanbetakentonewalevelwithtest-drivendevelopment(TDD).

InTDDtestsare,asthenameimplies,createdbeforeactualproductioncode

whichforcesdeveloperstothinkaboutthedesignandtestability


100/602

ofthesystem.Asaresultthedesignisclearandnumberofdefectslow(Beck,2003).Fromtestingpointofviewthenice

thingaboutTDDisthatacomprehensiveregressiontestsetisgotas

abonus.Verypromisingresultsfromusingtest-drivendevelopmentarereportedforexample

byGerasetal.(2004)andMaximilienandWilliams(2003),


101/602

andmypersonalexperiencesarealsoextremelypositive.

Thescopeofthisthesisisonhighertestautomationlevelsthanunittesting.

ThatdoesnotmeanthatIwouldconsiderunittestingandunittest

automationlessimportant.Onthecontrary,Istronglybelievethatinmostcases

automationeffortsshouldbefirstdirectedintounittestinglevel


102/602

whereneededinvestmentsaresmallandlikelytopaybackfast.

ComponentTesting

Componentsvaryalotfromonesituation

toanotherbutroughlyspeakingacomponentisacollectionofrelatedunits

whichhaveacommoninterfacetowardsothercomponents.Theseinterfacesmaybesimilar

programminginterfacesasinunitlevelbuttheycanalso


103/602

beimplementedwithmanyhigherleveltechniqueslikeCOM,XMLandHTTP.

Automationincomponentlevelisnormallynottoodifficult.Test

toolscanbehookedintothesameinterfacewhichisusedbyother

components.Inmanycasestherealsoexistsgoodandfreetoolsfordriving

theinterface(e.g.HTTPUnit)andevenifnoready-madetool


104/602

isavailabledeveloperscaneasilywriteone.(Pettichord,2002)

SystemTesting

Thedifferencebetweenacomponentandasystemis

thatsystemsfunctionstandalonewhilecomponentsareonlyusedaspartofa

system.Inreallifethisdifferencecanberathervagueassystemsare

oftenintegratedtogethertoformevenbiggersystems.


105/602


106/602


Systemshavedifferentinterfacestowardstherestoftheworlddependingonwhethertheyaregoingtobe

usedbyhumansorothersystems.Systemsusedbyhumanshavesomekind

ofuserinterface,eithergraphicalornon-graphical,andsystemsusedbyothersystems

havesimilarinterfacesascomponents.Graphicaluserinterfacesarenotoriously


107/602

hardtoautomatebutotherinterfacesaregenerallyrathereasy.(Pettichord,2002)

1.2.4TestingActivitiesSoftwaretestingismuchmorethatjust

executingtestcasesandsimilarlytestautomationisnotlimitedtoautomatingonly

testexecution.

DesigningTestCases

Functionalsystemandacceptance

testsaredesignedbytestengineersusingsystemrequirementsand


108/602

formaltestdesigntechniqueslikeequivalencepartitioningandboundaryvalueanalysis.Designinggoodtestcasesisnoteasyanditisoneofthe

mainskillsaprofessionaltestengineermustpossess.(FewsterandGraham,1999;Craig

andJaskiel,2002;Burnstein,2003)

Therearealsowaystoautomate

thetestdesignprocess.Expectedresultsfortestscansometimes


109/602


110/602

basedtestingthesystemismodeledinsuchadetailthattestcasescanbederivedautomaticallyfromthemodel(Dalaletal.,1999;

FewsterandGraham,1999;Robinson,2000).Plentyofothermethodslikeadata

miningapproach(Lastetal.,2003),agoal-drivenapproach(Memonetal.,1999)

andarequirement-basedtestgeneration(Tahatetal.,2001)are


111/602

alsopossible.

Thesekindofmethodsareveryinterestingbutscopeofthisthesisissolelyontestexecutionandtaskrelated

toit.Itshould,however,bepossibletointegrateatestdesignautomation

tooltoatestexecutionframeworkpresentedinlaterchapters.


112/602


ExecutingTestCasesandAnalyzingResults

Aftertestcaseshavebeendesignedandcreatedtheycan

beexecutedandresultsverifiedasillustratedinFigure1.1.Testsareoften

re-executedwhenanewversionofthetestedsystemisavailable.Automatingthese

regressiontestsoratleastanexemplarysubsetofthem,


113/602

oftencalledsmoketests,canmaketestexecutionandresultanalysisconsiderablyfaster.(Zambelich,1998;Pettichord,1999)

ReportingTestResults

Aftertestengineershaveruntheirteststheyreportfindingstotheproject

teamandmanagement.Iftestexecutionisfullyautomateditmakessenseto

automatealsoreporting.Itprobablyisnotagoodidea


114/602

tosendalltestreportsautomaticallytomanagersmailboxesbutifatestreportiscreatedautomaticallytestengineersdonotneedto

spendtimegatheringinformationfromtestlogsandelsewhere.

TestManagement

Manytestmanagementtaskslikeplanning,monitoring,schedulinganddefecttracking

canbesupportedbetools.Thesetoolsandautomationpossibilities


115/602


116/602

results.(Zallar,2001)

ToolAidedTesting

Smallscaletestautomationjusthelpsmanualtesting.Testautomationtoolsareusedin

areaswherecomputersarebetterthanhumansandwheretoolsareavailableor

easilyimplemented.Forexampletaskslikecheckingthataninstallationcopiedallfiles

torightplacesandverifyingthattwofileshavesame


117/602

datataketimeandareerrorpronetohumansbuteasilyautomated.ThisautomationapproachisstronglyadvocatedbyBach(2003).Henotesthat

insmallscaleautomationthetestcodeis


118/602


generallysosimpleandinexpensivethatitcaneitherbeeasilyfixedortrashedifitisbroken

bychangesinthetestedsystem.

Bach(2003)proposesanidea

thatsocalledtoolsmithsworkwithtestengineersandprovidethemwithtools

andutilitiesbasedontheirneeds.Toolsmithsshouldhavegood


119/602


120/602

toolsbuiltbydevelopersfortheirdevelopmenttimetesting.Bach(2003)andKaneretal.(2001)listmoretoolsandsourceswheretofind

them.AsKaneretal.(2001)putsit,youmayalreadyhavemore

testtoolsthanyourealize.

Eventhoughthescopeofthis

thesisisinlargescaletestautomationIbelievesmall


121/602

scaletestautomationcanoftenbeabetterstrategybecauseitdoesnotrequirebiginvestmentsandprovideshelpfast.Especiallyifriskwith

largerframeworksfeeltoobigitisbettertostartsmall,gainmore

experienceandthenlaterinvesttolargerframeworks(Zallar,2001;Bach,2003).

TestAutomationFrameworks

Whentestautomationis


122/602

takentohighestleveltestscanbestartedwithapushofabutton,theycanbeleftrunningover-nightunattendedandnextmorning

testresultsarepublished.Thiskindofautomationclearlyrequiressomekindof

asystemwhichmakescreating,executingandmaintainingtestcaseseasy.Thesystem

shouldprovidesomecorefunctionality(e.g.monitoringandreporting)and


123/602


124/602

areunstructured,havetestdataembeddedintothescriptsandthereisnormallyonescriptperonetestcase.Scriptsaremainlygeneratedusing

captureandreplaytoolsbutmayalsobemanuallycoded.Thiskindof

scriptisvirtuallynon-maintainableandwhenthetestedsystemchangestheyneedto

becapturedorwrittenagain.


125/602


2.Insecondgenerationscriptsarewell-designed,modular,robust,documentedandthusmaintainable.Scriptsdonotonlyhandle

testexecutionbutalsoforexamplesetupandcleanupanderrordetectionand

recovery.Testdataisstillembeddedintothescripts,though,andthereis

onedriverscriptsperonetestcase.Codeismostly


126/602

writtenmanuallyandbothimplementationandmaintenancerequireprogrammingskillswhichtestengineersmaynothave.3.Thirdgenerationframeworkshaveallthesame

goodcharacteristicsalreadyfoundfromsecondgeneration.Theygoforwardbytakingtest

dataoutofthescriptswhichhastwosignificantbenefits.Firstbenefitis

thatonedriverscriptmayexecutemultiplesimilartestcases


127/602

byjustalteringthedataandaddingnewtestsistrivial.Secondbenefitisthattestdesignandframeworkimplementationareseparatetasksformercan

begivensomeonewiththedomainknowledgeandlattertosomeonewithprogramming

skills.Thisconceptiscalleddata-driventesting.Keyword-driventestingtakestheconcepteven

furtherbyaddingkeywordsdrivingthetestexecutingintothe


128/602


129/602

isfollowing.

Designingandimplementingalargescaletestautomationframeworkforfunctionaltestingincomponentandsystemlevel.Theframeworkmust

beabletoexecutetests,verifyresults,recoverfromexpectederrorsandreport

results.Itmustalsobebotheasytouseandmaintain.Itdoes

notneedtoprovidesupportforautomationoftestplanning


130/602

anddesign.

Knowinghowtoautomatesomethingisclearlynotenoughtomakeanautomationprojectasuccessandsomeofthe

otherimportantthingstotakeintoaccountarelistedinTable1.3.These

issuesareexcludedfromthescopeofthisthesisbuttheyarelisted

hereforcompletenesssake.Theyalsohelptounderstandhow


131/602

manyissuestherearetohandlewhenconductingtestautomationprojectsandhowbigandchallengingtheseprojectscanbe.


132/602

CHAPTER1.INTRODUCTION12Whentoautomate.Toolselectionprocess.Whattoautomate.Takingtoolsandframeworksintouse.Whattoautomatefirst.

Testplanninganddesign.Whatcanbeautomated.Effecttodevelopmentandtesting

processes.Howmuchtoautomate.Otherorganizationalissues.Buildorbuy.Costsand

savings.Buildin-houseoruseconsultants.Metrics.

Table


133/602

1.3:Someimportantissuesexcludedfromthescope

1.4MethodologyHevneretal.(2004)assertthatsoftwareengineeringresearchischaracterizedby

twocomplementaryparadigms:behavioralscienceanddesignscience.Behavioralscienceistheoreticaland

hasitsoriginsinnaturalsciencewhiledesignscienceoriginatesfromengineeringand

triestofindpracticalsolutionsforresearchproblemsfromthe


134/602

field.Theimportanceofpracticalresearchandconcrete,evaluatedresultsisemphasizedbothbyHevneretal.(2004)andShaw(2002)andthemethodology

ofthisthesisisbasedonmodelspresentedbythem.

Much

oftheknowledgeofthisthesiscomesfrommultipletestingandtestautomation

booksandarticleslistedinthebibliography.Thisisnot


135/602

onlyaliteraturestudy,however,asIhavesixyearsexperiencefromtestingandtestautomationusingdifferenttoolsandtechniques.Thisexperiencegives

firsthandknowledgeaboutthesubjectandalsogreatlyhelpsprocessingtheinformation

readfromtheliterature.

1.5GoalsTheobjectiveofthethesis

ispresentingaconceptforalargescaletestautomation


136/602


137/602

feasibilityoftheframeworkbasedonthem.Adaptrequirementsbasedonthenewinformationifnecessary.


138/602


139/602

howtomeettheserequirements.Chapter3isthecoreofthethesis.InthatchapteraframeworkmeetingtherequirementssetinChapter

2isintroducedandexplainedindetail.

InChapter4the

presentedframeworkistestedagainstdefinedrequirements.Thisisdonebyimplementinga

prototypeoftheframeworkandusingitintwosimple


140/602

butnon-trivialpilotstestingaWindowsapplicationandawebpage.

ResultsandexperiencesfromthepilotarecollectedtogetherinChapter

5.Basedontheresultsthegeneralfeasibilityofthesuggestedframeworkis

evaluatedandpossiblychangesmadetotherequirementsetdefinedearlier.

Chapter6isleftforconclusions.Thereitisevaluated


141/602

howwellrequirementsforthisthesis,definedinSection1.5,aremet.


142/602

Chapter2

RequirementsforTestAutomationFrameworks

Chapter1introduceddifferentviewstotestautomationandstatedthatthe

scopeofthisthesisislargescaletestautomationframeworks.Theaimof

thechapteristodefinerequirementsfortheseframeworksanddiscussmethodologiesand

techniquesneededtomeettherequirements.

2.1High


143/602

LevelRequirementsAshortlistofhighlevelrequirementsapplicableforalllargescaletestautomationframeworks,derivedfromKit(1999)andothersources,

isshowninTable2.1.

Automatictestexecutionisahigh

levelfunctionalrequirementfortestautomationframeworks.Howitcanbeaccomplishedand

howitbreaksintomoredetailedrequirementsisdiscussedin


144/602

Section2.2.Easeofuseandmaintainabilityarenonfunctionalintheirnatureandingeneralachievingthemisharderthansimplyimplementingthefunctional

requirements.Sections2.32.8presentmethodsandtechniqueshowtheycanbefulfilledand

howtheyturnintomoredetailedrequirements.Detailedrequirementsarefinallysummarizedin

Section2.9.

2.2FrameworkCapabilitiesThissectionexplains


145/602

whatcapabilitiesatestautomationframeworkmusthave.Discussedfeaturescanbeseenasfunctionalrequirementsforautomationframeworks.

14


146/602


147/602

tousebytestengineersoritisverylikelytobeabandoned.Frameworkusersmustbeabletodesignandedittests,run

themandmonitortheirstatuseasilywithoutanyprogrammingskills.(Kaner,1997;Kit,

1999;Nagle,2000)

MaintainabilityItmustbeeasyandfastto

maintainbothtestdataandframeworkcodewhenthetested


148/602


149/602

pushofabuttonandruntestsonitsown.Thismeansthatframeworkmustbeabletosetupthetestenvironmentand

preferablyalsocheckthatallpreconditionsaremet.(FewsterandGraham,1999;Pettichord,

1999)

2.2.2StartingandStoppingTestExecutionItmustbepossible

tostarttestexecutionmanually.Itisalsoconvenientif


150/602


151/602

canbeimplementedsimilarlyusingexternaltools.

2.2.3HandlingErrorsPartofrunningtestsunattendedisrecoveringfromerrorscausedbythe

testedsystemorthetestenvironmentnotworkingasexpected.Testframeworkought

tonoticeerrorsituationsandcontinuetestingwithoutmanualintervention.Handling


152/602

CHAPTER2.REQUIREMENTS

allpossiblebutrareerrorsisseldomworththeeffort,though,andover-engineeringshouldbeavoided.(Pettichord,1999)

2.2.4VerifyingTestResultsAnintegralpartoftestexecutionis

verifyingtestresults.FewsterandGraham(1999)defineverificationasoneormore

comparisonsbetweenactualoutcomeofatestandpredefinedexpected


153/602

outcome.Theyalsoexplaindifferentcomparisontechniqueswhichareoutofthescopeofthisthesis.

2.2.5AssigningTestStatusAftera

testisexecutedanditsresultsverifieditoughttobegivena

status.Ifthetestwasexecutedwithoutanyproblemsandallcomparisonsbetween

actualandexpectedoutcomesmatchthetestgetsapass


154/602

status.Ineveryothercasethestatusisfail.Besidesthestatuseverytestcaseshouldalsogetashortbutdescriptivestatusmessage.

Forpassedteststhismessageisnormallynotthatimportantbutwithfailed

testsitcangivedetailsaboutthecauseoftheproblem(e.g.Calculation

failed:expected3,got2orInitializingtestenvironmentfailed:


155/602


156/602

HandlingExpectedFailuresThemostfrustratingpartoffailureanalysisisgoingthroughtestcaseswhichareknowntofailandcheckingwhetherthey

havefailedsimilarlyasbeforeorhavenewdefectsappeared.Thatiswasteful

andclearlysomethingthatshouldbeautomatedasFewsterandGraham(1999)recommends.

Tobeabletodifferentiateexpectedfailuresfrom


157/602

newonestheframeworkmustknowwhatistheexpectedoutcomewhentestfails.Thismeansthattheframeworkmuststoreboththeexpected

outcomeandtheexpectedfailedoutcomeofthetest.Whenatestfails,

andisexpectedtodoso,theexpectedfailedoutcomecanbe


158/602


Figure2.1:Handlingexpectedfailures

comparedagainsttheactualoutcomeasseenintheFigure

2.1.Ifoutcomesmatchanexpectedfailureisdetected,otherwiseanewdefect

hasbeenfound.Irecommendcommunicatingexpectedfailuresusingstatusmessagesbut,as

alreadydiscussed,itcanbearguedthataspecifictest


159/602

statusshouldbeusedinstead.IneithercaseitsureisfasttoanalyzefailedtestwhichrightawaytellssomethinglikeExpectedfailure.

Defectid:#2712.

FewsterandGraham(1999)alsorecommendnotifyingtest

engineerswhenatestwhichisexpectedtofailpasses(i.e.defecthas

beenfixed).Thiswaytheycanmanuallyverifythatthe


160/602

testreallyhaspassedanddefectisfixedwithoutsideeffects.Aftereverypassedtesttheframeworkshouldcheckwhetherthetestwasexpected

tofailbycheckingdoesithavetheexpectedfailoutcome.Iftest

wasexpectedtofailitshouldbelabeledaccordinglyIrecommendsettingthestatus

messagetosomethinglikeDefect#2712fixed.Thewholeprocess


161/602


162/602


FailUsedtologthestatusofafailedtestcase.PassUsedtologthestatusof

apassedtestcase.FatalUsedwhenanunrecoverablefailurepreventingfurthertest

executionoccurs.Thiskindofproblemismostofteninthetestenvironment

(e.g.diskfullornetworkdown).ErrorUsedwhenan


163/602

unexpectederrorpreventingcurrenttestsexecutionoccurs.Forexamplethebuttontopushinatestdoesnotexists.FailureUsedwhenthetest

outcomedoesnotmatchtheexpectedoutcome.WarningUsedwhenarecoverableerror

notaffectingtestexecutionoccurs.Forexampledeletingsometemporaryfilefails.Info

Usedtolognormalpassedverifications.testexecutionlikestarting


164/602

testandDebugGivesdetailedinformationaboutwhatthetestautomationframeworkoraparticulartestcaseisdoing.Usedmainlyfordebuggingthe

frameworkortestcases.TraceSimilartodebugbutcontainsevenmoredetailed

information.

Table2.2:Suggestedlogginglevels

testexecutionprocess

andhowthetestedsystembehavedareneeded.Ontop


165/602

ofthattheframeworkmustlogwhatitisdoinginternallytomakeiteasiertodebugproblemsintheframeworkitself.

Themaindilemmawithanykindoflogging,asstatedbyPettichord(2002),

ishowmuchtolog.Loggingtoomuchmakesfindingrelevantinformationhard

andhugelogfilescauseastorageproblem.Thenagain


166/602

loggingtoolittlecanmakelogstotallyuseless.Thereisnoperfectsolutionforthisproblembutforexampleusingdifferentlogginglevelsmake

theproblemsmaller.

Multiplelogginglevelsprovideamechanismforcontrolling

howmuchdetailiscaptured.Forexamplelowestlevels,thosethatproducemost

output,caninnormalcasesbefilteredoutbeforethey


167/602

areevenlogged.Ifdetailedinformationisneededthefiltercanbeadjusted.Laterwhenlogsareviewedentriescanbefilteredevenfurther

sothatonlyinterestinglevelsareviewed.Whatlevelsareneededdependsultimately

onthecontextbutIhavefoundthelistinTable2.2pretty

goodformostsituations.Twoexampleshowtheselevelscan


168/602

beusedarefoundinFigure2.2.


169/602


TimestampLevelMessage2005050215:14:01InfoTestexecutionstarted.SUT:Calculatorv.0.6.1,Testsuite:Calculatorbasic20050502

15:14:04InfoTestenvironmentsetupwithoutproblems2005050215:14:04InfoStartingtestcase

Add012005050215:14:05InfoTest1+2returned32005050215:14:05

PassAdd012005050215:14:06InfoStartingtestcaseAdd


170/602


171/602

+0.5returned12005050215:14:09PassAdd042005050215:14:10InfoTestexecutionended.4/4testcasesexecuted(100%),3/4passed(75

%),1/4failed(25%).

TimestampLevelMessage2005050216:23:31Info

Testexecutionstarted.SUT:Calculatorv.0.6.2,Testsuite:Calculatorbasic2005050216:23:43

FatalSettinguptestenvironmentfailed2005050216:23:43InfoTest


172/602


173/602

thethresholdlevelarenottobelogged.Adjustingthelevelmustbeeasyforanyoneexecutingtestsanditshouldpreferablybepossible

alsowhiletestsarerunning.Thedefaultlogginglevelshouldnormallybeinfo.

Infolevelgivesrequiredinformationabouttestexecutionbutnodebuggingdetailswhich

shouldnotbeneedednormally.ForexamplelogsinFigure


174/602

2.2wouldbetoouninformativeifhigherlevelwasusedandunnecessarilylongiflevelwaslower.


175/602


2.2.8AutomaticReportingTestlogshavealltheinformationfromtestexecutionbut,especiallyiftheyarelong,

theyarenotgoodforseeingteststatusesataglance.Thiskind

ofviewisprovidedbyconcisetestreports.Testreportsprovidestatisticalinformation

aboutthequalityofthetestedsystemandtheyare


176/602

notonlyimportantfortestengineersbutalsofordevelopers,managersandbasicallyeveryoneinvolvedwiththeproject.(FewsterandGraham,1999)

Reportsshouldnotbetoolongordetailed.Havingashortsummaryand

listofexecutedtestcaseswiththeirstatusesisprobablyenough.Listof

itemseverytestreportshouldhaveispresentedbelowadaptedfrom


177/602

Buwaldaetal.(2002)andFigure2.3showsasimpleexample.

Nameandversionofthetestedsystem.Identificationofthe

executedtestsuite.Thetotalnumberofpassedtestsandexecutedtests.

Listoferrorsfound.Optionallyalistofallexecutedtests

alongwiththeirstatuses.Testreportscanbeeithercreated


178/602


179/602


180/602


181/602


182/602


183/602

functionsand,asillustratedinFigure2.4,theyinteractdirectlywiththetestedsystem.Simplelinearscriptsarefasttowriteinitiallyandthey

arethuswellsuitedforsmalltasks.Theproblemwithlinearscriptsis

thatwhentasksgetbiggerandmorecomplicatedalsotheygrowlonger,more

complicatedandgenerallyhardtomaintain.Theyalsotendto


184/602

haverepeatingcodewhichcouldbetakenoutandreused.Ifmanysimilartestscriptsarecreatedthemaintenanceproblemjustgrowsbiggerbecause

onechangeinthetestedsystemcanrequirechangesineveryscript.(Fewster

andGraham,1999)


185/602


2.3.2TestLibrariesandDriverScriptsStructuredcodingisacommonprogrammingparadigmanditisofcourse

applicablewithtestcodetoo(Kelly,2003).Inlowlevelitcanmean

simplyusingsmallfunctionsinthesamefileasthemaintestscript.

Inhigherlevelusetestfunctionsareplacedtoexternal,


186/602

sharedtestlibrariesfromwheretheycanbeusedbyanytestscript.

Whentestlibrariesdomostoftheactualtesting

worktestscriptscanbeshortandsimple.Becausescriptsareonlydriving

thetestexecutiontheyarecustomarilycalleddriverscripts.Figure2.5showshow

twodriverscriptsusessametestlibrarytointeractwith


187/602


188/602


189/602


190/602

of,documenting,naming,andstoringofcreatedscriptsisabigtask.Theyalsonotethatifreusablefunctionsarenoteasytofind

andwelldocumentedautomatorsarelikelytowritetheirownversions.Marick(1997)

notesthatwritingwell-designedAPItoatestlibraryisashardas

writinganygoodAPIanditshouldnotbeexpected


191/602


192/602

frameworks.Icannotseehowmaintainabilitycouldbeachievedwithoutthem.Librariessureareanoverkillforsimplecasesbutinthosecasesthe

wholeideaofbuildingabigframeworkisquestionable.

2.4Data-Driven

Testing2.4.1IntroductionSimpletestscriptshavetestdataembeddedintothem.This

leadstoaproblemthatwhentestdataneedsto


193/602

beupdatedactualscriptcodemustbechanged.Thismightnotbeabigdealforthepersonwhooriginallyimplementedthescriptbut

foratestengineernothavingmuchprogrammingexperiencethetaskisnot

soeasy.Ifthescriptislongandnon-structuredthetaskishard

foreveryone.Anotherproblemwithhavingthetestdatainside


194/602

testscriptsisthatcreatingsimilartestswithslightlydifferenttestdataalwaysrequiresprogramming.Thetaskmaybeeasyoriginalscriptcanbecopied

andtestdataeditedbutatleastsomeprogrammingknowledgeisstillrequired.This

kindofreuseisalsoproblematicbecauseoneparticularchange


195/602


inthetestedsystemmayrequireupdatingallscripts.(Strang,1996;Nagle,2000)

Becauseofthese

problemsembeddingtestdataintoscriptsisclearlynotaviablesolutionwhen

buildinglargertestautomationframeworks.Abetterapproachisreadingthetestdata

fromexternaldatasourcesandexecutingtestbasedonit.


196/602


197/602

2005)

2.4.2EditingandStoringTestDataBecausedata-driventestdataistabularitisnaturaltousespreadsheetprogramstoedit

it.Testengineersandbusinesspeopleare,asMugridgeandCunningham(2005)point

out,alsolikelytobefamiliarwithspreadsheetprogramsandalreadyhavethem

available.Anotherbenefitisthatspreadsheetprogramsareoftenused


198/602

forsimpletestmanagementtasksandinthatcasethereislessneedforstoringsamedataindifferentplacesorcopyingitback-and-forth.

Spreadsheetfilescanbesavedincomma-separated-values(CSV),tab-separatedvalues(TSV)or

spreadsheetprogramsnativeformats.CSVandTSVfilesarehandybecausetheyare

veryeasytoparsebutunfortunatelymostspreadsheetprogramsseem


199/602

toalterthedatawheytheyopenthesefiles.ForexampleifIstorephonenumber+358912345andversionnumber1.5.1toaCSV

fileandopenittomyExceltheyareauto-correctedto358912345and

1.5.2001,respectively.Theeasiestsolutionforthisproblemisstoringdatausingprograms

nativeformatandonlyexportingitintoCSVorTSV.


200/602

Anotherpossibilityisprocessingthenativeformatdirectlybutitrequiressomemoreinitialeffort.

HTMLtablesprovidedanothereasy-to-parseformatfor

presentingtestdata.EditingHTMLwithadecentgraphicaleditorisnearlyas

convenientasusingaspreadsheetprogrambutunliketheirnativeformatsHTMLcan

alsobeeditedwithanytexteditorifnothingelse


201/602


202/602

theymaynotbeenough.Therealsolutionforscalabilityproblemsisstoring


203/602


Figure2.6:Data-drivenapproach

Figure2.7:Data-driventestdatafile

data=

open(testdata.tsv).read()lines=data.splitlines()[1:]#[1:]excludestheheaderrow

forlineinlines:testId,number1,operator,number2,expected=line.split(

\t)#Actualtestfunctionalityexcluded

Listing


204/602


205/602


testdataintoacentraldatabase.Thatwaythetestdatacanbeeditedusingforexamplea

web-basedinterfaceanddriverscriptscanquerythedatadirectlyfromthedatabase.

Buildingsuchasystemisabigprojectitselfandcanberecommended

onlyifscalabilityisarealproblem.

Bird


206/602


207/602

wouldthenagainratherselectadatabasewithaweb-baseduserinterfacebecauseitwouldalsoprovideacentralizeddatastorage.

2.4.3

ProcessingTestDataImplementingascriptforparsingdata-driventestdatacanbe

surprisinglyeasywithmodernscriptinglanguages.ForexampledatainFigure2.7can

beexportedintoaTSVfileandparsedwithfour


208/602


209/602


210/602

isrequiredfromtheparseritshouldbeimplementedasacommonmodulewhichalldriverscriptscanusethiswasalreadyseeninthe

Figure2.6.Howtheparsershouldbeimplementedandwhatkindoffunctionality

itneedsisfurtherdiscussedinSection3.2andChapter3ingeneral.

2.4.4PromisesandProblemsMainbenefitofdata-driven


211/602

testautomationisthatitmakescreatingandrunninglotsoftestvariantsveryeasy(Strang,1996;FewsterandGraham,1999;Kaneretal.,

2001).Editingtestsandaddingnewsimilaronesiseasyandrequiresno

programmingskills(Kaner,1997).Anotherbenefitisthattestdatacanbedesigned

andcreatedbeforetestimplementationorevenbeforethetested


212/602


213/602


(Strang,1996;Pettichord,2003)anditisusableinmanualtestingevenifautomationimplementationisneverfinished.

Overall,thedata-drivenapproachisthekeyforease-of-useinlargescaletest

automation.

Thedata-drivenapproachalsohelpswithmaintenance.Whenthetested

systemchangesitisoftenpossibletochangeonlyeither


214/602

thetestdataorthetestcodeandtheirmaintenanceresponsibilitiescanalsobedividedbetweendifferentpeople.(Strang,1996;Kaner,1997;Marick,1997;

Kaneretal.,2001;Rice,2003)

Thebiggestlimitationofthe

data-drivenapproachisthatalltestcasesaresimilarandcreatingnewkinds

oftestsrequiresimplementingnewdriverscriptsthatunderstanddifferent


215/602

testdata.ForexampletestdatainFigure2.7anditsparsingscriptinListing2.1aredesignedfortestingcalculationswithonlytwo

numbersandwouldrequiremajorchangestohandlelongertestslike5*

8+2=42.Ingeneraltestdataanddriverscriptsare

stronglycoupledandneedtobesynchronizedifeitherchanges.


216/602

Anotherdisadvantageofdata-driventestingistheinitialset-upeffortwhichrequiresprogrammingskillsandmanagement.(FewsterandGraham,1999)

2.5Keyword-Driven

Testing2.5.1IntroductionPrevioussectionintroduceddata-driventestingandstatedthatithas

multiplepromises.Italsomentionedthatitsbiggestlimitationisthatalltest

casesaresimilarandcreatingtotallynewtestsrequiresprogramming


217/602

effort.Asolutionforthislimitation,offeredbyFewsterandGraham(1999)andKaneretal.(2001)amongothers,isthekeyword-drivenapproachwhere

notonlythetestdatabutalsodirectivestellingwhattodowith

thedataaretakenfromtestscriptsandputintoexternalinputfiles.

Thesedirectivesarecalledkeywordsandtestengineerscanuse


218/602

themtoconstructtestcasesfreely.Thebasicideareadingtestdatafromexternalfilesandrunningtestsbasedonitstaysthesameasin

data-driventesting.AsFewsterandGraham(1999)putit,keyword-driventestingisa

logicalextensiontodata-driventesting.Exampleofakeyword-driventestdatafileis

seeninFigure2.8.

2.5.2EditingandStoring


219/602


220/602


221/602


222/602

beimplementedonlyonceandplacedintoacommon,reusablemodule.

Afterthetestdataisparsedthedriverscriptmustbe

abletointerpretkeywordsandexecutethespecifiedactionusingassignedarguments(Fewster

andGraham,1999).Tokeeptheframeworkmodularitisagoodidea

tohaveahandler(i.e.atestfunction)foreach


223/602


224/602

theinterfacelevel.Whentestinghigherlevelfunctionalitylikebusinesslogiclowlevelkeywordstendtomaketestcasesverylongandhigherlevel

keywords(e.g.Addinourcalculatorexample)aremuchmoreusable.Sometimesit

ispossibletouseonlyeitherloworhighlevelkeywordsandbuild

keywordhandlersaccordingly.Oftenbothlevelsareneededandthen


225/602

itisagoodideatoconstructhigherlevelkeywordsfromlowlevelkeywords.ForexamplelowlevelkeywordsInputandPushcouldbe

usedtocreatehigherlevelkeywordsAdd,Subtract,MultiplyandDivideandsimilarly

EqualscouldbecreatedusingPushandCheck.Figure2.10showshowthese

higherlevelkeywordsmaketestcasesshorter.

A


226/602


227/602

createnewones.

Amoreflexiblesolutionisallowingtestengineerstoconstructhigherlevelkeywordsusingthesameinterfacetheyuse

fordesigningtestcases.Thesekindoftechniquesarepresentedwithdifferentimplementation

bothbyBuwaldaetal.(2002)andNagle(2000).Figure2.12showsone

optionhownewkeywordscouldbeconstructed.To


228/602


229/602


Figure2.10:Keyword-driventestdatafilewithhigherlevelkeywords

Figure2.11:Creatinghigher

levelkeywordsintestlibrary

Figure2.12:Creatinghigherlevel

keywordsintestdesignsystem


230/602


differentiatethesenewkeywordsfromthoseimplementeddirectlyinthetestlibraryIliketocallthemuser

keywordsandbasekeywords,respectively.Testlibrariesdonothavehandlersforuser

keywords,insteadtheyareexecutedbyexecutingbasekeywords,orotheruserkeywords,

theyareconstructedfrom.

Themainbenefitin


231/602

makingitpossibletoconstructnewkeywordsusingthetestdesignsystemisthatnewkeywordscanbecreatedandmaintainedeasilywithoutprogramming

skills.Ofcourseitcanbearguedthatcreatinguserkeywordsisalready

programmingbutatleastitissoeasythateveryoneinvolvedwithtest

automationcanlearnit.Thebiggestproblemwiththeapproach


232/602

isthatprocessingtestdatagetsmorecomplicatedbut,onceagain,thesmallincreaseininitialeffortisinsignificantwhenbuildingalargescale

automationframework.ImplementationissuesarediscussedingreaterdetaillaterinChapter3

andespeciallyinSection3.3.

2.5.5PromisesandProblemsKeyword-driventesting

hasallthesamebenefitsasdata-driventesting.Asalready


233/602

mentionedthemainadvantageoverdata-driventestingisthatthekeyword-driventechniquemakesiteasyevenfornon-programmerstocreatenewkindsoftests.

ComparingFigures2.7and2.8provethispointprettyexplicitly.Keyword-driventestingis

abigstepforwardfrompuredata-driventestingwherealltestsaresimilar

andcreatingnewtestsrequiresnewcodeintheframework.


234/602


235/602

keywords.Themainproblemofkeyword-driventestingisthatitrequiresamorecomplicatedframeworkthandata-driventesting.

2.6OtherImplementationIssues

Sameprocesses,methodologiesandtechniqueswhichareimportantwhendesigningandimplementingany

non-trivialsoftwaresystemareimportantalsowhenbuildinglargertestautomationframeworks(Zallar,

2001).ThoseimplementationissueswhichIfeelarethemost


236/602

importantarediscussedinthissection.


237/602


2.6.1ImplementationLanguageProgramminglanguagesareneededwhenbuildingtestautomationtoolsandframeworks.Oftenautomationtoolsare

alsocontrolledusinginputscriptswhichhavetheirownlanguage.Requirementsforlanguage

usedwhenbuildingatoolandforlanguageusedwiththetoolcan

beverydifferent.Withmultipleprogramminglanguagesavailablepickingthe


238/602

rightchoiceindifferentsituationsisnotaneasytask.Selectingalanguagealreadyusedintheorganizationorknownotherwiseisalwaysa

safebetbutitisnottheonlycriteria.

SystemProgramming

Languages

SystemprogramminglanguagessuchasC++andJavaaredesigned

forbuildingdatastructures,algorithmsandlargesystemsfromscratch


239/602

(Ousterhout,1998).Theyareoptimizedforperformanceandscalabilityratherthanforfastprogrammingspeed.Theselanguagesmaywellbeusedwhenbuildingtest

automationtoolsandframeworksespeciallywhendevelopersknowthemwellalreadyanddo

notknowsuitablehigherlevelscriptinglanguages.Foralanguagetobeused

asaninputscriptforatooltheyaretoo


240/602

lowlevel.

ScriptingLanguages

ScriptinglanguagessuchasPerl,Python,Ruby,TclandVisualBasicarehigherlevellanguagesthan

systemprogramminglanguages.Ousterhout(1998)statesthatinsteadofbuildingthingsfromscratch

theyaredesignedforgluingcomponentsandsystemstogether.Ousterhout(1998)suggeststhat

programmingsomethingtakesfivetotentimeslesscodein


241/602

scriptinglanguagethaninsystemprogramminglanguage.Becauseprogrammerswriteapproximatelythesamenumberoflinesinsametimeregardlessthelanguage,scriptinglanguages

mayfastendevelopmenttimewiththesamemagnitude.Shorterprogramsarealsoeasier

tomaintain.

Ousterhout(1998)mentionsthatscriptinglanguagesaresomewhatslower

thansystemprogramminglanguages,butcurrentcomputersaresofast


242/602

thatitisnormallynotimportant.Pettichord(1999)notesthatintestautomationthisslightperformancedifferenceishardlyeveraproblembecausemost

ofthetimeisspentwaitingforthetestedsystemtorespondfor

inputsorsettinguptestenvironment.

Ihaveexperienceusingboth

systemprogrammingandscriptinglanguagesfortestautomationandI


243/602

cansecondallthepositivethingsOusterhout(1998)saysabout


244/602


245/602

files.Shellscriptsarenotasflexibleashigherlevelscriptinglanguagesbuttheyareveryhandyforsmalltasks.Iuseshellscripts

regularlyforsmallautomationtasksbutIconsiderthemtoocumbersomeforimplementing

largerframeworks.

VendorScripts

Vendorscriptsareproprietarylanguages

implementedbytoolvendorstobeusedwiththesetools.


246/602

Testtoolsusingvendorscriptshouldbeapproachedwithacertaincautionforreasonslistedbelow.

Nobodyknowsthem.Developersand

sometestengineersarelikelytoknowmanyprogramminglanguagesbutnotlikely

toknowsomespecialvendorscript.Hiringpeoplewithpreviousknowledgeishard.

(Pettichord,2001)Notmatureenough.Vendorscriptsdonot


247/602

havethosefeaturesmatureprogramminglanguagesandtheirsupportlibrarieshave.Ifvendorscriptdoesnothaveforexamplegoodenoughstringhandlingcapabilities

youhavetoimplementthemyourself.(Pettichord,2001)Vendorscriptshavecompatibility

problemswithothertoolsandlanguages.(Rice,2003)Becauseofthesereasonsmy

adviceisavoidingvendorscriptsaltogetherifpossible.Luckilythey


248/602

aregettingrarenowadaysasvendorshavestartedembeddingexistingmaturescriptinglanguagestotheirproducts.ExamplesofthisareMercurysQuickTestProwhich

usesVisualBasicandPushToTestsTestMakerusingJython(aJavaimplementationofthe

Pythonlanguage).


249/602


2.6.2ImplementationTechniqueManualProgramming

Manualscriptwritingisanormalprogrammingtaskandifdone

properlyscriptsareeasytouseandmaintain.Ithastwoobviousdrawbacks:

writingscriptstakessometimeandrequiresprogrammingskillswhichtestengineersmay

nothave.(FewsterandGraham,1999;MosleyandPosey,2002)


250/602

CaptureandReplay

Captureandreplaytoolsaresupposedtoovercomeproblemswithmanualscriptwriting.Theideaisthat

youpusharecordbuttonandletthetoolrecordeveryactionkeystroke,mouse

movementorclickwhileexecutingtestmanually.Thetoolstorescapturedactionsina

scriptforlaterplayback.Thisapproachsoundsattractiveand,as


251/602

Meszaros(2003)note,itmaybesuitableforsituationswherescriptsareneededfast,theycanbetrashedinthefutureandprogrammingskills

requiredformanualscriptingarenotavailable.

Thebiggestproblemwith

captureandreplayisthatmaintainingrecordedscriptsisnearlyimpossible.Moreproblems

associatedwithitarelistedforexamplebyKaner(1997),


252/602


253/602


254/602

isoutofthescopeofthisthesisbutthissectiongoesthroughfewaspectsIfeelmostimportantwhentalkingaboutlargertest

automationframeworks.


255/602


VersionControl

Versioncontrolandconfigurationmanagementarerequiredforanylargersoftwareprojectandtest

automationframeworksarenoexception.Maintainingalltheneededtestwareanditsdifferent

versionsisanightmarewithoutanadequateversioncontrolsystem.(FewsterandGraham,

1999;Zallar,2001;Buwaldaetal.,2002)

Coding


256/602

andNamingConventions

Codingandnamingconventionscovereverythingfromvariable,functionandfilenamingtocodeindentation.Theideaofcoding

conventionsisthatwheneveryonewritessimilarcodeitistheneasyto

understandandmaintainbyanyone.Consistentnamingofmodules,functionsandfilesgreatly

helpswithreusewhenmoduleandfunctionnamesareeasy


257/602

torememberandevenguess.(FewsterandGraham,1999)

Thesekindsofconventionareveryimportantwhenanythinglargerisprogrammedand

shouldnotbeforgottenwhenbuildinglargetestautomationsystems.Conventionsshouldbe

agreedanddocumentedwhenstartingatestautomationproject.Oftenthisiseasy

asconventionsusedwhenimplementingtestedsystemscanbeused


258/602

straightawayorwithslightmodifications.

Documentation

Lackofadequatedocumentationmakesusingtestlibrarieshardbecausepeoplenewto

themneedtospenttimefindinghowandunderwhatcircumstancestouse

them.Documentationisalsoneededinmaintenance.Needforqualitydocumentationiscommonly

understoodbuttherearetwoproblems:firstthedocumentationmust


259/602

bewrittenandthenitmustbekeptupdated.(FewsterandGraham,1999)

Forcodedocumentationpurposessemi-automaticdocumentationsystemssuchas

Javadocaregettingmoreandmorepopular.Withthesetoolstheoriginaldocumentation

iskeptwithinthecodesothatitisavailablewhencodeis

readandeditedanditcaneasilybekeptupdated


260/602

withthecode.Documentationcanbeextractedwithspecialtoolssothatitisreadablewithawebbrowserorsomeotherviewer.Tools

canalsoreadfunctionandvariablenamesfromthecodesothatthey

donotneedtobedocumentedmanually.

Codedocumentationisvery

importantbutnotenoughonitsown.Otherneeded


261/602


262/602


documentsincludeinstructionsforsettingup,usingandmaintainingtheframework.Itisthequalityofthedocumentation,

notitsquantity,thatisimportant.(FewsterandGraham,1999)

2.7

TestabilityMostoftenthehardestpartofatestautomationeffortisintegrating

theselectedtoolwiththetestedsystem.Pettichord(2002)and


263/602

Kaneretal.(2001)statethatinsteadoffinetuningthetooltohandleallpossiblequirksofthetestedsystemitmightbe

bettertoinvestonincreasingthetestabilityofthesystem.Pettichord(2002)has

noticedthatthesuccessofthewholeautomationprojectmaywelldependon

testabilityissuesandalsoothers,e.g.FewsterandGraham(1999),


264/602

Kaneretal.(2001)andBach(2003),underlineimportanceoftestability.Testabilityproblemsaresimilarinsmallandlargescaletestautomationprojectsbut

inlargerscaleitspositiveeffecttomaintenancegetsmoreandmoreimportant.

Pettichord(2002)definestestabilityasvisib

Documents

Data-Driven and Keyword-Driven Test[1]