Picture Courtesy of: Dr. Sarah Ford (a.k.a. Mrs. Matthew ... · How I Learned to Stop Worrying...

HowILearnedtoStopWorrying&LovetheBug

Picture Courtesy of: Dr. Sarah Ford (a.k.a. Mrs. Matthew Hertz)

¨ Cannotprovetherearenobugs¤ Canonlyshownobugsexistonthosetests

HowToWriteTests

Testingshowsthepresence,nottheabsence

ofbugs

¨  20%ofcodehas80%ofbugs¤ Modulesthataremostcomplex,intricate,ordetailed¤ Locationswhereexpectationsofdatamightdiffer¤ Codeintransition:frequentlychangedmodules¤ Anyplacewhereprogramrequiresuserinput

¨  Focustestingeffortstoconcentrateonthesebugs¤ Tests(&testing)expensive&simplertoolsforeasycode¤ Automationmatters;errorsoftenoccuratjoinpoints

WhatToTest&Why

Assumetheworst:Focustestingon

unlikelysituations

TestsKeyConcept

InputTests…

…andFinally

¨  Inrealworld,somecasesmaynotbeworthtesting¤ Mustassumebugsexistsoideallytesteverything¤ Savetime,donotchangeinputtochecksameidea¤ Simplegetters&setterseasy,butcheckbeforecommit¤ Focusonpossibilities,donotcheckimpossiblecases

WheretheBugsAren’t

GoodTests

public class Stock { private double cost; //Constructor&gettersimple&skippedforspace

//Decreasescostofastock;deltaismax.dropincost //Returnsupdatedvalueofcost public double reduceCost(double delta) { }

//Evenmorecodewouldbehere,werethisnotanexampleforclass

public class StockTest { @Test public void t1() { Stock ibm = new Stock(141.31); assertEquals(141.31, ibm.reduceCost(0), 0.001); assertEquals(141.31, ibm.getCost(),0.001); }

MoreGoodTests

public class StockTest { @Test public void t2() { Stock siri = new Stock(7.10); assertEquals(0.0, siri.reduceCost(8), 0.001); assertEquals(0.0, siri.getCost(),0.001); }

public class Stock { private double cost; //Constructor&gettersimple&skippedforspace

//Decreasescostofastock;deltaismax.dropincost //Returnsupdatedvalueofcost public double reduceCost(double delta) { }

//Evenmorecodewouldbehere,werethisnotanexampleforclass

NotaGoodTest

public class StockTest { @Test public void t3() { Stock htz = new Stock(15.09); assertEquals(?????, htz.reduceCost(-100), 0.001); assertEquals(?????, htz.getCost(),0.001); }

public class Stock { private double cost; //Constructor&gettersimple&skippedforspace

//Decreasescostofastock;deltaismax.dropincost //Returnsupdatedvalueofcost public double reduceCost(double delta) { }

//Evenmorecodewouldbehere,werethisnotanexampleforclass

NotaGoodTest

public class Stock { private double cost; //Constructor&gettersimple&skippedforspace

//Differencefromcostatwhichpeoplesoldstock;deltaismax.dropincost //Returnsupdatedvalueofcost public double reduceCost(double delta) { }

//Evenmorecodewouldbehere,werethisnotanexampleforclass

public class StockTest { @Test public void t3() { Stock htz = new Stock(15.09); assertEquals(?????, htz.reduceCost(-100), 0.001); assertEquals(?????, htz.getCost(),0.001); }

TestsKeyConcept

Assumetheworst:Focustestingon

unlikely(butNOTimpossible)situations

¨  Smallbugsinloopscancreatehugeerrors¤ Lotsoftimeexecutingincreasesoddsofhittingrarecase¤ Oftenerroronlyappearswhenresultsused,notinloop¤ Debuggingoftentricky,sincemanyscenariostotestout

¨  Runoften+hard-to-debug==criticaltotestwell¤ Findingbugsimportant,sincequalitydependsonthis¤ Knowingbugsexistsuseless;mustalsosimplifyfixes¤ Sonarrowingbug'scausejustasneededasfindingbug

LoopTestingImportant

TypesofLoops

¨  Forallsimpleloops,tryinputsthat:¤ Skiploopentirely¤ Make1passthroughtheloop¤ Make2passesthroughtheloop¤ Makempassesthroughtheloop,where(m>2)

¨  Ifloopexecutedatmostntimes,tryinputsthat:¤ Maken-1&npassesthroughtheloop

SimpleLoop

¨  Firsttestsetrunsallouterloopsexactlyonce¤  Innerloopruns(min+1),average,(max-1)&maxtimes

¨  Thenrunallbuttwoinnermostloopsexactlyonce¤  Innerloopsrun(min+1),average,(max-1)&maxtimes

¨  Testsshouldcontinuegrowingloop-by-loop

NestedLoops

TypesofLoops

ConcatenatedLoops

¨  Ifloopsareentirelyindependent¤ Noconditions,variables,orvaluesincommon¤ Woo-hoo!Justperformsinglelooptestsoneach

¨  Otherwisetreatasnestedloops&makelifeeasier¤ Workasifthefirstloopistheoutermostloops

UnstructuredLoops

¨  Figureouttheprocessleadingtothisdecision¤  Burnartifactsandcoderesultinginthisabomination

UnstructuredLoops

¨  Figureouttheprocessleadingtothisdecision¤  Burnartifactsandcoderesultinginthisabomination¤ Anyoneinvolvedshouldterminatedimmediately

UnstructuredLoops

¨  Figureouttheprocessleadingtothisdecision¤  Burnartifactsandcoderesultinginthisabomination¤ Anyoneinvolvedshouldterminatedimmediately

¨  ReWrite“missing”documents,startingfromscratch

¨  Unittestsgoodforsometasksworkingonback-end¤ Butwhatabouttasksimplementingfront-endcode?¤ Userwantsresultsandonlyknowswhattheycansee¤ Correctresultsimpossibleifback-endfailsunittests

¨  Back-endcodeveryimportantsocannotskiptests¤ Butinvisibletouserandclientdoesnotcareaboutcode

Back-EndTesting

¨  Needtotestfront-endtasksthatdisplayinformation¤ GUIclassescanbecheckedagainstuserstories¤ JUnittestcaseslessusefulperformingthesetests¤ Automationlackshumantouch;cannotcheckaesthetics

Front-EndTesting

¨  Worstapproach:Clickingaround&seewhatbreaks¤ Simple&fast,butmaynotdiscoveractualcauseofbugs¤ Unrepeatable&slowwhencheckingentiresystem¤ Donebydevelopers,tendstofollowexpecteduses

Front-End"Testing"

¨  Betterapproach:Step-by-stepscripttestsforerrors¤ Lowoverhead&simple,butalsoeasytoforgettorun¤ Discoverunexpectedbugsbyhavingtestersrunscripts¤ Goodrules-of-thumbexisttofindmanycommonerrors¤ ListintaskinZenHub;manywantfilestoholdscripts

Front-EndTesting

¨  Betterapproach:Step-by-stepscripttestsforerrors¤ Lowoverhead&simple,butalsoeasytoforgettorun¤ Discoverunexpectedbugsbyhavingtestersrunscripts¤ Goodrules-of-thumbexisttofindmanycommonerrors¤ ListintaskinZenHub;manywantfilestoholdscripts

Front-EndTesting

¨  Bestapproach:AutomatetestingwithUItool/code¤ Createssetupcosts,butguaranteespredictableresults¤ Cancompensateforloadtimes&otherrealissues¤ Oftenincludebothprogramming&scriptingsetups

ValidationTesting

¨  Allowsautomatedtestingofweb-basedapplications¨  Testsuitereportsresultsofrunning1ormoretests¨  Oftencreatemanytestcases;eachexposes1bug

¨  AddtestsinJava/C#/PythonwithWebDrivermodule¤ ManylanguageshaveSeleniumlibrariestodrivetests¤ Loadspage&definesAPIusedtoevaluateitscontents

¨  IfusingIDE,abletocreate&runsinbrowser¤  IDEeasiertouse:canrecordactionsinbrowserastest¤ WillalsoallowupdatingorrewritingSelenesescript

Selenium

¨  Easiestmoduletouse,butneedsChrome*towork¤ DownloadviaChromeWebStoretobereadytouse¤ Scriptsmostlyrecordedbyclickingonelementstotest

¨  Startprocessusingthecommandopentoloadpage¤ click[AndWait]"clicks"onitemthatyouidentify¤ Scriptcanalsoentertextintoelementusingtype

¨  LikexUnittests,reliesonassertionstodefinechecks¤ assertTitlecheckstitleofpage(textshownontab)¤ CheckiftextonpageusingverifyTextPresent ¤ verifyElementPresentchecksifelementonpage

SeleniumIDE

public static void main(String[] args) { WebDriver driver = new EdgeDriver(); driver.get("http://www.google.com"); WebElement el = driver.findElement(By.name("q")); element.sendKeys("Hawaiian-Print Computer"); element.submit(); WebDriverWait stall = new WebDriverWait(driver, 10); boolean result = stall.until(

new ExpectedCondition<Boolean>() { public Boolean apply(WebDriver d) { return d.getTitle().startsWith("Hawaiian"); }}); System.out.println("Met expectations: " + result); driver.quit(); }

SeleniumWebDriver+Java

driver = webdriver.Firefox() driver.get("https://cse.buffalo.edu/~mhertz") assert "Matthew Hertz" == driver.title crselnk = driver.find_element_by_xpath( "/html/body/table[2]/tbody/tr/td[1]/p/a") crselnk.click() result = WebDriverWait(driver, 10).until( lambda x : "CSE442" in x.title) assert result

SeleniumWebDriver+Python

¨  Inputoverflows:Typestringlongerthannormal/fits¤ Checkthattextisaccepted(orprovidesGOODerror)¤  Iftextisaccepted,areresultsreadableorusable?¤ Specifytext(trymanysizes)inscriptscheckingthis

CommonFront-EndErrors(1)

¨  Structureoverflow:Makepanelslargerthanpage¤ Doesthiscreateerrorsorissystemabletohandledata¤ Doitemsresize,scroll,orprovidewaytoseeeverything?¤ Similartolooptests;detailinginputstousecritical

CommonFront-EndErrors(2)

¨  ViolateAssumptions:Assumeusersjerks(ordumb)

CommonFront-EndErrors(3)

¨  ViolateAssumptions:Assumeusersjerks(ordumb)¤ Theywillmakeworstchoice.Howdoessystemreact?¤ Whatifneededfilesdeleted,networklost,orsimilar?¤ Scriptexplainhowtostart&whaterrorshouldbeshown

CommonFront-EndErrors(3)

¨  DuplicationIssues:Repeatedlyentersameinput¤  IfWhentheyadd/removemultipletimes,whatiserror?¤ Doesapphandle(orprovideclues)forimpatientusers?

CommonFront-EndErrors(4)

¨  DuplicationIssues:Repeatedlyentersameinput¤  IfWhentheyadd/removemultipletimes,whatiserror?¤ Doesapphandle(orprovideclues)forimpatientusers?¤ “Back”buttontempting,whatdoesitdotowebapp?

CommonFront-EndErrors(4)

¨  InvalidData:Intentionallyforceinvalidresults¤ Scriptactionscreatingillegalstateinwidgetattheend¤ Feb29remaininnon-leapyears?Movestartafterend?¤ Giveintothedarkside&tricksystemintobadstates

CommonFront-EndErrors(5)

¨  Resizingissues:Canithandledifferentwindowsizes¤ Scriptresizingwindow&makesureprogramstillusable¤ Setmonitortosmallerscreen&seeiflayoutworks¤ Tryforcingscrollbaruse&seehowuserswillreact

CommonFront-EndErrors(6)

¨  JavaScriptisahorrible,horriblelanguage

Non-UIJavaScriptTesting

¨  JavaScriptisahorrible,horriblelanguage¤ BrendanEichcreated&implementedJSin10days¤ Namewasbuzzword;neverrelatedtoJava¤ Notintendedasstandard;NetscapelookingtobeatIE¤  "Standards"exist,butimplementationsvarygreatly

¨  Languagecombinesmanyfeaturestoprovideitall¤ OO,functional,ordeclarativecodesupportexists¤ JSoftenimplementsmiddle-tier&front-endlayers¤ Testdifferentlevelsseparatelywouldbeideal,buthow?

Non-UIJavaScriptTesting

¨  RealweaknessofJSisdifficultyintestingcode¤ Notgreatlyused,butMochabestunittestinglibrary¤ Middle-tiertestedviafront-endinmany/mostsituations¤ Nothelpfultounderstand&fix,butshouldfindbugs

¨  GoodlanguagecompilingtoJSisalternateapproach¤ Oncecomplete,usetestsandtoolsfororiginallanguage¤ ScalaJS,TypeScript,&Dartdevelopedforthispurpose¤ Allofthisalsoassumesthatbugnotcreatedbycompiler

Non-UIJavaScriptTesting

Utility:Isituseable

OtherTestingIssues

Utility:IsituseableReliability:Willyouendupleadstoryonnightlynews?

OtherTestingIssues

Utility:IsituseableReliability:Willyouendupleadstoryonnightlynews?Robustness:Howlongofdisclaimerwillitneed?

OtherTestingIssues

Utility:IsituseableReliability:Willyouendupleadstoryonnightlynews?Robustness:Howlongofdisclaimerwillitneed?Performance:WillitfinishbeforeBuffalowinsatitle?

OtherTestingIssues

KeyPoint

=

¨  "Acceptancetests"checkuserstorycomplete¤ Ensuresfeatureworksandreadyfordeployment¤ Runbyclientsohavetobescriptedtests¤ Ensurefullunderstandingofallaspectsoffeature

¨  "Tasktests"checkthatataskiscomplete¤ Ensurestaskcompleteandreadyforinclusion¤ Runbydeveloperssocanbescriptedtestsorunittests¤ Findsbugsduringcoding&throughlaterchanges¤ Alsodefineswhatsuccessliketoenableparallelwork

Terminology

¨  WorkonSprint1¤ Rememberthetests!Testsvalidateyourunderstanding

ForNextLecture