TestsAntipatterns
by Maciej Przewoźnik
AgendaSlow TestOveruse of mocksObscure Test
Agenda (2)Manual TestFragile TestErratic TestEager Test
Agenda (3)Verbose TestMystery GuestTest Code DuplicationHigh Test Maintenance Cost
Slow Test
Slow TestStabilizing a very slow testIndividual slow testsExecuting all tests is too long
Slow TestStabilizing a very slow testIndividual slow testsExecuting all tests is too long
- -
- -
Slow Test - stabilizing a very slowtest
Don't steer a rover on Mars from EarthExecute a test ...8 minutes later: failureFix & rerun10 minutes later: failureFix & rerun10 minutes later: failure
Slow Test - stabilizing a very slowtest
What to do in the meantime?GTD: don't multitaskCPU cache vs "brain cache"Context switch cost
Slow Test - stabilizing a very slowtest
What to do in the meantime?Solution: use the fastest machineHave the machine in the companyOther tests on slower machines
Slow Test - stabilizing a very slowtest
What to do in the meantime?For performance or stress tests with bigamount of dataSolution: use smaller data sets forstabilization firstThen run with bigger datasets
Slow Test - stabilizing a very slowtest
Solution: use REPL environment tostabilize it fasterReal-eval-print loopA failure shouldn't invalidate previous results!
Slow Test - stabilizing a very slowtest
Solution: stabilize parts separatelyGiven a test:
Generate a big file.Consume it.
Don't remove the file if failure occurs in (2).(to be continued)
Slow TestStabilizing a very slow testIndividual slow testsExecuting all tests is too long
Slow Test - individual tests areslow
What is a slow test?a developer gets irritated or bored waiting forfinishis tempted to get a break, surf the Internet,chat, or walk around
Slow Test - individual tests areslow
What is a slow test?Sometimes more than a second, sometimes amillisecondNot long individually, but all tests execution istoo longOr a group of tests is too long
Slow Test - individual tests areslow
Impact:Developers stop running them after every codechangeDevelopers wait for a coffee break, lunch or ameeting to run themDelayed feedback, loss of "flow"Decreases productivity and job satisfaction
Slow Test - individual tests areslow
Cause: Slow dependent component (1)Solution: Fast Fake ObjectsReal DB → Fake DB (e. g. In Memory DB)Real Cloud → In memory cloudFilesystems: HDD, SSD → RAM disk:
mount t tmpfs o size=200m tmpfs /mnt/tmp
Slow Test - individual tests areslow
Cause: Slow dependent component (2)Example: Backup applicationsSolution: Recorded testRecord operations done by backupapplicationsWarning: Recorded test may be a Fragile Test
Slow Test - individual tests areslow
Cause: Slow environment or layerSolution: Separate tested logic fromenvironment:
Hardware/GUI/HTTP layerThreads, processesClouds, etc.Kernel code
Slow Test - individual tests areslow
Cause: Over-engineered fixtureLong setup of many complicated objectsEven if most of them are not neededOr using full product in testsSolution: General Fixture → MinimalFixture
Slow Test - individual tests areslow
Cause: Manual testsSlow by definitionEven slower if repeated"Please check that the system just works"before a releaseAutomated tests may get written anyway later
Slow Test - individual tests areslow
Cause: Manual testsOften under pressure or under "give-it-to-me-now driven-development"No future returnsMay be worthless after just a single commitLack of tests automation will slow testdevelopment later
Slow Test - individual tests areslow
"Just Say No to More End-to-End Tests"Testing pyramid
Manual tests
End-to-end tests
Integration tests
Component tests
Fast Unit Tests
Slow TestStabilizing a very slow testIndividual slow testsExecuting all tests is too long
Slow Tests - executing all tests istoo long
In some big IT companies:build time: one minuterelease to production: 8 minutesnot just a release – a release to production!
What problems may it address?
Slow Tests - executing all tests istoo long
Impact: Developers context-switching,worse productivity
tests start ...15 min: a test failsfix & rerun20 min: a test fails
Slow Tests - executing all tests istoo long
Impact: Integrating code slowerBigger release queueSlower features delivery, missing deadlines
Slow Tests - executing all tests istoo long
Impact: Production big fixes aredelayed
A customer requires a fix, we need to test itRelease a quick fix or a well tested fix?
Slow Tests - executing all tests istoo long
Cause: Too much overlap betweentests
Test1: [ DATA LOADING ][doSth][check]
Test2: [ DATA LOADING ][doSth][check]
Test3: [ DATA LOADING ][doSth][check]
Slow Tests - executing all tests istoo long
Cause: Too much overlap betweentests
Use Shared Fixture?Shared setup for many testsWarning: may get closer to Erratic Tests,
High Test Maintenance Cost
Slow Tests - executing all tests istoo long
Cause: Too Many TestsToo many tests run too frequentlySolution: subsets of tests can be run atthe timeAll tests within larger period of timePrerelease tests: all with fast fakesPostrelease tests: with real objects
Slow Tests - executing all tests istoo long
Cause: Too Many TestsThe system is too largeSolution: break it into independentsubsystems
Slow Tests - executing all tests istoo long
Cause: Slow MachinesOnly one integration serverToo slow integration serversInsufficient parallelization: look at "top","vmstat"
Overuse of (strict)mocks
Overuse of mocksImpact:Tests can be harder tounderstand
The extra code detracts from the tester authorsintent
Overuse of mocksImpact: Tests are harder to maintain
Focus: implementation vs public interfaceFocus: implementation vs behaviorExtra code for mock behavior neededOver-specified softwareFragile
Overuse of mocksImpact: Less assurance that the codeis working properly
It's hard to guarantee that the mocks behave asreal implementationsEspecially over time
Overuse of mocksWhen the real object cannot be used:
They are too slowTheir setup is too complexThey charge money, etc.
Overuse of mocksAlternatives:
Fake objectsHermetic local servers (for cash transactions,etc.)
Consider thefollowing bad code ...
class TeaHouse(object): def __init__(self, cash_service, tax_service): self.cash_service = cash_service self.tax_service = tax_service self.money_total = 0
def order_cup(self, credit_card): cup = Tea() self.cash_service.begin_transaction() result = self.cash_service.charge( credit_card, cup.price) if result: self.money_total += cup.price self.cash_service.commit_transaction() self.tax_service.register_transaction( cup.price ) else: self.cash_service.cancel_transaction()
def should_compute_doubled_price_for_two_cups(): cs = mock(CashService) cs.expect_call("begin_transaction").times(2) cs.expect_call("charge").times(2).repeatedly( Return(Cup()) ) cs.expect_call("end_transaction").times(2) ts.mock(TaxService) ts.expect_call("register_transaction").times(2) t = TeaHouse(cs, ts) c = CreditCard(money=usd(30)) t.set_tea_price(usd(10)) t.order_cup(c) t.order_cup(c) assert t.money_total == usd(20)
Overuse of mocksWe actually don't care for interaction with cashservice and tax service here
Overuse of mocksAlternative: use a fake
def should_compute_doubled_price_for_two_cups(): t = TeaHouse( FakeInMemoryCashService(), FakeTaxService() ) c = CreditCard(money=usd(30)) t.set_tea_price(usd(10)) t.order_cup(c) t.order_cup(c) assert t.money_total == usd(20)
Overuse of mocksAlternative: use a stub or a non-strictmock:
def should_compute_doubled_price_for_two_cups(): t = TeaHouse( StubCashService(always_ok=True), StubTaxService() ) c = CreditCard(money=usd(30)) t.set_tea_price(usd(10)) t.order_cup(c) t.order_cup(c) assert t.money_total == usd(20)
But in this particularsituation, even better
is ...
One function, oneresponsibility
Overuse of mocksclass TeaHouseOrders: def __init__(self): # no cash_service and no tax service! self.orders = [] def order_cup(credit_card): self.orders.append(TeaOrder()) return def total_price(): return sum([o.price() for o in self.orders])
Overuse of mocksclass TeaHouseCash: def charge(price): self.cash_service.begin_transaction() result = self.cash_service.charge(cc, price) if result: self.cash_service.commit_transaction() else: self.cancel_transaction() return self.cups
Overuse of mocksdef should_compute_doubled_price_for_two_cups(): t = TeaHouseOrders() t.set_tea_price(usd(10)) t.order_cup() t.order_cup() assert t.total_price() == usd(20)
Neither mocks nor fakes needed!Important concept: separate descriptionfrom execution
Obscure Test
Obscure TestImpact:
Harder to understandHarder to maintainDevelopers will not read them asdocumentationMay lead to Can result in a Buggy Test, and
then Production Bugs
Obscure TestGood Tests:
Tests should act as documentationAnd also a self-verifying executablespecification
Obscure TestIs it a good test?
def should_have_the_same_permissions_after_copy(): fs = FileSystem("rw") f = File("a.txt", "rwxrwxrwx") fso = FileSystemOps(fs) fso.copy(f, "b.txt") assert(fso.fileperm("a.txt") == fso.fileperm("b.txt"))
Obscure TestIs it a good test?
def should_have_the_same_permissions_after_copy(): fs = FileSystem("rw", "noauto", 620, false, true, true, 8096, 10*1024*1024*1024, 1000000) f = File("a.txt", "rwxrwxrwx", 64, "xyzxyz", "20161212", "20161212", "20161212") fso = FileSystemOps(fs, true, false, 1024, 11) fso.copy(f, "b.txt") assert(fso.fileperm("a.txt") == fso.fileperm("b.txt"))
Obscure TestCause: Irrelevant information
which values affect the outcome?There can be a hundred constants in a testcode
Obscure TestCause: Too much informationSolution: introduce higher-level testlanguage
def file_copy_permissions_should_be_as_original(): given_a_filesystem() given_a_file_with_permissions("rwxrwxrwx") when_a_file_is_copied() then_the_new_file_has_permissions("rwxrwxrwx")
Obscure TestCause: Too much information, EagerTest
The test verifies too much functionality in asingleTest Method and therefore is hard tounderstand
Obscure TestCause: Too much information, EagerTest
def test_my_filesystem(): fs = FileSystem("rw", "noauto", 620, false, true, true, 8096, 10*1024*1024*1024, 1000000) assert(fs.isReadOnly() == false) f = File("a.txt", "rwxrwxrwx", 64, "xyzxyz", "20161212", "20161212", "20161212") assert(f.getPermissions() == "rwxrwxrwx") assert(f.getCreationDate() == "20161212") fso = FileSystemOps(fs, true, false, 1024, 11) fso.copy(f, "b.txt") assert(fso.fileperm("a.txt") == fso.fileperm("b.txt")) assert(fso.size("a.txt") == fso.size("b.txt")
Obscure TestCause: Too much information, EagerTest
Often a Fragile TestAnd a High-Maintenance Test
Obscure TestCause: Too much information
Not caring about clean codeNo refactoring"These are only tests""just do it in-line" mentality
Obscure TestCause: Too little information – MysteryGuest
Cause and effect not clearSomething is hidden from the Test Method
Obscure TestCause: Too little information – MysteryGuestDoesn't exist because is copied to root?
def test123(): fs = createFileSystem() f = new File("a.txt", "rwxrwxrwx", 64*MB, "xyzxyz", "20161212", "20161212", "20161212" ) fso = new FileSystemOps(fs, true, false, 1024, 11) fso.copy(f, "/b.txt") assert(fso.exists("b.txt") == false)
Obscure TestCause: Too little information – MysteryGuestNo, because createFileSystem creates read-onlyfilesystem:
def createFileSystem(): return new FileSystem( "ro", "noauto", 620, false, true, true, 8096, 10*1024*1024*1024, 1000000 ))
Obscure TestCause: Setup-action-verify phases notclear
def test123(): fs = createFilesystem() f = createFile("a.txt") fso = createFso() f2 = fso.copy(f, "b.txt") s1 = fso.size("a.txt") s2 = fso.size("b.txt") assert(s1 == s2)
Do we test "size" here or "copy"?Solution: Given-when-then
Obscure TestCause: Setup-action-verify phases notclearSolution: Given-when-then:
given_filesystem() given_file() given_file_system_operations() when_the_file_is_copied() then_the_new_file_size_is_as_original()
Obscure TestCause: Overcomplicated Fixture
class TestFSFixture: def createRealOnlyFS(): def createRWFS(): def createJournalingFS(): def createTransactionalFS(): def createNFS(): def createHDFS(): def createNoAcFs(): def createTwoFilesystems(): def createPerfOptFS(): def createBuggyFS(): def createSlowFS():
Obscure TestCause: Overcomplicated FixtureThe fixture is too big and too complexTo understand the test, reading the fixture maybe necessarySolutions:
Divide a General Fixture into smaller FixturesUse Minimal FixtureUse Facades
Obscure TestCause: Indirect Testing
Testing one component through othercomponentsExample: Execute a product in debug mode tosee if log rotation worksRoot Cause: software not designed fortestability
SourcesGerard Meszaros - xUnit Test Patternstesting.googleblog.comPaul Chiusano and Rúnar Bjarnason - FunctionalProgramming in Scala
THE END