Tests antipatterns

TestsAntipatterns

by Maciej Przewoźnik

AgendaSlow TestOveruse of mocksObscure Test

Agenda (2)Manual TestFragile TestErratic TestEager Test

Agenda (3)Verbose TestMystery GuestTest Code DuplicationHigh Test Maintenance Cost

Slow Test

Slow TestStabilizing a very slow testIndividual slow testsExecuting all tests is too long


- -

- -

Slow Test - stabilizing a very slowtest

Don't steer a rover on Mars from EarthExecute a test ...8 minutes later: failureFix & rerun10 minutes later: failureFix & rerun10 minutes later: failure


What to do in the meantime?GTD: don't multitaskCPU cache vs "brain cache"Context switch cost


What to do in the meantime?Solution: use the fastest machineHave the machine in the companyOther tests on slower machines


What to do in the meantime?For performance or stress tests with bigamount of dataSolution: use smaller data sets forstabilization firstThen run with bigger datasets


Solution: use REPL environment tostabilize it fasterReal-eval-print loopA failure shouldn't invalidate previous results!


Solution: stabilize parts separatelyGiven a test:

Generate a big file.Consume it.

Don't remove the file if failure occurs in (2).(to be continued)


Slow Test - individual tests areslow

What is a slow test?a developer gets irritated or bored waiting forfinishis tempted to get a break, surf the Internet,chat, or walk around


What is a slow test?Sometimes more than a second, sometimes amillisecondNot long individually, but all tests execution istoo longOr a group of tests is too long


Impact:Developers stop running them after every codechangeDevelopers wait for a coffee break, lunch or ameeting to run themDelayed feedback, loss of "flow"Decreases productivity and job satisfaction


Cause: Slow dependent component (1)Solution: Fast Fake ObjectsReal DB → Fake DB (e. g. In Memory DB)Real Cloud → In memory cloudFilesystems: HDD, SSD → RAM disk:

mount t tmpfs o size=200m tmpfs /mnt/tmp


Cause: Slow dependent component (2)Example: Backup applicationsSolution: Recorded testRecord operations done by backupapplicationsWarning: Recorded test may be a Fragile Test


Cause: Slow environment or layerSolution: Separate tested logic fromenvironment:

Hardware/GUI/HTTP layerThreads, processesClouds, etc.Kernel code


Cause: Over-engineered fixtureLong setup of many complicated objectsEven if most of them are not neededOr using full product in testsSolution: General Fixture → MinimalFixture


Cause: Manual testsSlow by definitionEven slower if repeated"Please check that the system just works"before a releaseAutomated tests may get written anyway later


Cause: Manual testsOften under pressure or under "give-it-to-me-now driven-development"No future returnsMay be worthless after just a single commitLack of tests automation will slow testdevelopment later


"Just Say No to More End-to-End Tests"Testing pyramid

Manual tests

End-to-end tests

Integration tests

Component tests

Fast Unit Tests


Slow Tests - executing all tests istoo long

In some big IT companies:build time: one minuterelease to production: 8 minutesnot just a release – a release to production!

What problems may it address?


Impact: Developers context-switching,worse productivity

tests start ...15 min: a test failsfix & rerun20 min: a test fails


Impact: Integrating code slowerBigger release queueSlower features delivery, missing deadlines


Impact: Production big fixes aredelayed

A customer requires a fix, we need to test itRelease a quick fix or a well tested fix?


Cause: Too much overlap betweentests

Test1: [ DATA LOADING ][doSth][check]




Cause: Too much overlap betweentests

Use Shared Fixture?Shared setup for many testsWarning: may get closer to Erratic Tests,

High Test Maintenance Cost


Cause: Too Many TestsToo many tests run too frequentlySolution: subsets of tests can be run atthe timeAll tests within larger period of timePrerelease tests: all with fast fakesPostrelease tests: with real objects


Cause: Too Many TestsThe system is too largeSolution: break it into independentsubsystems


Cause: Slow MachinesOnly one integration serverToo slow integration serversInsufficient parallelization: look at "top","vmstat"

Overuse of (strict)mocks

Overuse of mocksImpact:Tests can be harder tounderstand

The extra code detracts from the tester authorsintent

Overuse of mocksImpact: Tests are harder to maintain

Focus: implementation vs public interfaceFocus: implementation vs behaviorExtra code for mock behavior neededOver-specified softwareFragile

Overuse of mocksImpact: Less assurance that the codeis working properly

It's hard to guarantee that the mocks behave asreal implementationsEspecially over time

Overuse of mocksWhen the real object cannot be used:

They are too slowTheir setup is too complexThey charge money, etc.

Overuse of mocksAlternatives:

Fake objectsHermetic local servers (for cash transactions,etc.)

Consider thefollowing bad code ...

class TeaHouse(object): def __init__(self, cash_service, tax_service): self.cash_service = cash_service self.tax_service = tax_service self.money_total = 0

def order_cup(self, credit_card): cup = Tea() self.cash_service.begin_transaction() result = self.cash_service.charge( credit_card, cup.price) if result: self.money_total += cup.price self.cash_service.commit_transaction() self.tax_service.register_transaction( cup.price ) else: self.cash_service.cancel_transaction()

def should_compute_doubled_price_for_two_cups(): cs = mock(CashService) cs.expect_call("begin_transaction").times(2) cs.expect_call("charge").times(2).repeatedly( Return(Cup()) ) cs.expect_call("end_transaction").times(2) ts.mock(TaxService) ts.expect_call("register_transaction").times(2) t = TeaHouse(cs, ts) c = CreditCard(money=usd(30)) t.set_tea_price(usd(10)) t.order_cup(c) t.order_cup(c) assert t.money_total == usd(20)

Overuse of mocksWe actually don't care for interaction with cashservice and tax service here

Overuse of mocksAlternative: use a fake

def should_compute_doubled_price_for_two_cups(): t = TeaHouse( FakeInMemoryCashService(), FakeTaxService() ) c = CreditCard(money=usd(30)) t.set_tea_price(usd(10)) t.order_cup(c) t.order_cup(c) assert t.money_total == usd(20)

Overuse of mocksAlternative: use a stub or a non-strictmock:

def should_compute_doubled_price_for_two_cups(): t = TeaHouse( StubCashService(always_ok=True), StubTaxService() ) c = CreditCard(money=usd(30)) t.set_tea_price(usd(10)) t.order_cup(c) t.order_cup(c) assert t.money_total == usd(20)

But in this particularsituation, even better

is ...

One function, oneresponsibility

Overuse of mocksclass TeaHouseOrders: def __init__(self): # no cash_service and no tax service! self.orders = [] def order_cup(credit_card): self.orders.append(TeaOrder()) return def total_price(): return sum([o.price() for o in self.orders])

Overuse of mocksclass TeaHouseCash: def charge(price): self.cash_service.begin_transaction() result = self.cash_service.charge(cc, price) if result: self.cash_service.commit_transaction() else: self.cancel_transaction() return self.cups

Overuse of mocksdef should_compute_doubled_price_for_two_cups(): t = TeaHouseOrders() t.set_tea_price(usd(10)) t.order_cup() t.order_cup() assert t.total_price() == usd(20)

Neither mocks nor fakes needed!Important concept: separate descriptionfrom execution

Obscure Test

Obscure TestImpact:

Harder to understandHarder to maintainDevelopers will not read them asdocumentationMay lead to Can result in a Buggy Test, and

then Production Bugs

Obscure TestGood Tests:

Tests should act as documentationAnd also a self-verifying executablespecification

Obscure TestIs it a good test?

def should_have_the_same_permissions_after_copy(): fs = FileSystem("rw") f = File("a.txt", "rwxrwxrwx") fso = FileSystemOps(fs) fso.copy(f, "b.txt") assert(fso.fileperm("a.txt") == fso.fileperm("b.txt"))

Obscure TestIs it a good test?

def should_have_the_same_permissions_after_copy(): fs = FileSystem("rw", "noauto", 620, false, true, true, 8096, 10*1024*1024*1024, 1000000) f = File("a.txt", "rwxrwxrwx", 64, "xyzxyz", "20161212", "20161212", "20161212") fso = FileSystemOps(fs, true, false, 1024, 11) fso.copy(f, "b.txt") assert(fso.fileperm("a.txt") == fso.fileperm("b.txt"))

Obscure TestCause: Irrelevant information

which values affect the outcome?There can be a hundred constants in a testcode

Obscure TestCause: Too much informationSolution: introduce higher-level testlanguage

def file_copy_permissions_should_be_as_original(): given_a_filesystem() given_a_file_with_permissions("rwxrwxrwx") when_a_file_is_copied() then_the_new_file_has_permissions("rwxrwxrwx")

Obscure TestCause: Too much information, EagerTest

The test verifies too much functionality in asingleTest Method and therefore is hard tounderstand


def test_my_filesystem(): fs = FileSystem("rw", "noauto", 620, false, true, true, 8096, 10*1024*1024*1024, 1000000) assert(fs.isReadOnly() == false) f = File("a.txt", "rwxrwxrwx", 64, "xyzxyz", "20161212", "20161212", "20161212") assert(f.getPermissions() == "rwxrwxrwx") assert(f.getCreationDate() == "20161212") fso = FileSystemOps(fs, true, false, 1024, 11) fso.copy(f, "b.txt") assert(fso.fileperm("a.txt") == fso.fileperm("b.txt")) assert(fso.size("a.txt") == fso.size("b.txt")


Often a Fragile TestAnd a High-Maintenance Test

Obscure TestCause: Too much information

Not caring about clean codeNo refactoring"These are only tests""just do it in-line" mentality

Obscure TestCause: Too little information – MysteryGuest

Cause and effect not clearSomething is hidden from the Test Method

Obscure TestCause: Too little information – MysteryGuestDoesn't exist because is copied to root?

def test123(): fs = createFileSystem() f = new File("a.txt", "rwxrwxrwx", 64*MB, "xyzxyz", "20161212", "20161212", "20161212" ) fso = new FileSystemOps(fs, true, false, 1024, 11) fso.copy(f, "/b.txt") assert(fso.exists("b.txt") == false)

Obscure TestCause: Too little information – MysteryGuestNo, because createFileSystem creates read-onlyfilesystem:

def createFileSystem(): return new FileSystem( "ro", "noauto", 620, false, true, true, 8096, 10*1024*1024*1024, 1000000 ))

Obscure TestCause: Setup-action-verify phases notclear

def test123(): fs = createFilesystem() f = createFile("a.txt") fso = createFso() f2 = fso.copy(f, "b.txt") s1 = fso.size("a.txt") s2 = fso.size("b.txt") assert(s1 == s2)

Do we test "size" here or "copy"?Solution: Given-when-then

Obscure TestCause: Setup-action-verify phases notclearSolution: Given-when-then:

given_filesystem() given_file() given_file_system_operations() when_the_file_is_copied() then_the_new_file_size_is_as_original()

Obscure TestCause: Overcomplicated Fixture

class TestFSFixture: def createRealOnlyFS(): def createRWFS(): def createJournalingFS(): def createTransactionalFS(): def createNFS(): def createHDFS(): def createNoAcFs(): def createTwoFilesystems(): def createPerfOptFS(): def createBuggyFS(): def createSlowFS():

Obscure TestCause: Overcomplicated FixtureThe fixture is too big and too complexTo understand the test, reading the fixture maybe necessarySolutions:

Divide a General Fixture into smaller FixturesUse Minimal FixtureUse Facades

Obscure TestCause: Indirect Testing

Testing one component through othercomponentsExample: Execute a product in debug mode tosee if log rotation worksRoot Cause: software not designed fortestability

SourcesGerard Meszaros - xUnit Test Patternstesting.googleblog.comPaul Chiusano and Rúnar Bjarnason - FunctionalProgramming in Scala

THE END

Technology

Tests antipatterns