Test-Driven Development in (bed) with C
by James W Grenning - [email protected]
Bas Vodde - [email protected]
www.odd-e.comwww.renaissancesoftware.net
Presented at Agile 2011, Salt Lake City, UT, USA
and
Present
Scaling Lean & Agile Development
Thinking and Organizational Tools for Large-Scale Scrum
Craig LarmanBas Vodde
Practices for Scaling Lean & Agile
DevelopmentLarge, Multisite, and Offshore Products
with Large-Scale Scrum
Craig LarmanBas Vodde
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
This Work Flow is Designed to Allow Defects
2
Development
Test
Defects
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Edsger DijkstraThose who want really reliable software will discover that they must find means of avoiding the majority of bugs to start with, and as a result, the programming process will become cheaper. If you want more effective programmers, you will discover that they should not waste their time debugging, they should not introduce the bugs to start with.
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Can we Realize Dijkstra’s Dream andPrevent Defects with
Test Driven Development?
4
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Development and Test Work TogetherPreventing Defects
5
Development
Test
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
The Physics of Debug Later Programming (DLP)
• As Td increases, Tfind increases dramatically• Tfix is usually short, but can increase with Td
6
Bug discoveryMistake made(bug injection)
Bug found Bug fixed
Td Tfind T fix
Time
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
The Physics of Test Driven Development
• When Td approaches zero, Tfind approaches zero• In many cases, bugs are not around long enough to be considered
bugs.• See: http://www.renaissancesoftware.net/blog/archives/16
7
Mistake discovery
Mistake made
Root cause found
Mistake fixed
T d Tfind T fix
Time
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
What is TDD?
8
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
TDD Micro Cycle
• Write a test• Watch it not compile• Make it compile, but fail• Make it pass• Refactor (clean up any mess)• Repeat until done
9
The one rule: Only ever write code to fix a failing testwww.renaissancesoftware.net
[email protected] © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
TDD State Machine in C/C++
10
Write the test
Make the test link
Make the test compile
Compilation error
Link error
New test failsMake the test pass
Refactor(Make it right)
All tests pass
All tests pass No more tests
Choose a test
Start
Compilation error
Link error
DONE!
Programming error
Compiles Clean
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Test Driving a Circular Buffer (FIFO)
11
Out-Index In-Index
23 7 66 12 99 16 90
Out-Index In-Index
42 -6 23 7 66 12 99 16 90
Out-Index In-Index
99
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Create a Test-List(though, avoid analysis paralysis)
12
Circular Buffer Tests
Initially Empty Transition to empty Transition to Full Transition from Full Put-Get FIFO Put to full Get from empty Filled but not wrapped around Wrap around
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Think about the Solution(though, avoid analysis paralysis)
• We’ll need two indexes, one for putting integers in and one for taking them out
• We’ll need an array, that is dynamically sized during creation
• We’ll need wrap around logic
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Set up a Test Fixture
14
TEST_GROUP(CircularBuffer){ CircularBuffer* buffer;
void setup() { buffer = CircularBuffer_Create(); }
void teardown() { CircularBuffer_Destroy(buffer); }};
TEST(CircularBuffer, TestName){}
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Make a bare Skeleton of the Production Code
15
#ifndef D_CircularBuffer_H#define D_CircularBuffer_H
typedef struct CircularBuffer CircularBuffer;
CircularBuffer * CircularBuffer_Create();void CircularBuffer_Destroy(CircularBuffer *);
#endif // D_CircularBuffer_H
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Make a bare Skeleton of the Production Code
16
#include "CircularBuffer.h"#include <stdlib.h>
struct CircularBuffer{ int dummy;} ;
CircularBuffer* CircularBuffer_Create(){ CircularBuffer* self = calloc(1, sizeof(CircularBuffer)); return self;}
void CircularBuffer_Destroy(CircularBuffer* self){ free(self);}
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Choose a Test
17
Write the test
Make the test link
Make the test compile
Compilation error
Link error
New test failsMake the test pass
Refactor(Make it right)
All tests pass
All tests pass No more tests
Choose a test
Start
Compilation error
Link error
DONE!
Programming error
Compiles Clean
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Which Test?
18
Out-Index In-Index
41 59 14 33 7 31
In-Index Out-Index
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Production Code is Grown One Test at a Time
19
Code without
new feature
Code withnew tested
feature
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Write a Test
20
Write the test
Make the test link
Make the test compile
Compilation error
Link error
New test failsMake the test pass
Refactor(Make it right)
All tests pass
All tests pass No more tests
Choose a test
Start
Compilation error
Link error
DONE!
Programming error
Compiles Clean
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Start with a Test That is Easy to Get to Pass -- Designing the Interface
21
TEST_GROUP(CircularBuffer){ CircularBuffer* buffer;
void setup() { buffer = CircularBuffer_Create(); }
void teardown() { CircularBuffer_Destroy(buffer); }};
TEST(CircularBuffer, ShouldBeEmptyAfterCreate){ CHECK(CircularBuffer_IsEmpty(buffer));}
Write the test
Make the test link
Make the test compile
Compilation error
Link error
New test failsMake the test pass
Refactor(Make it right)
All tests pass
All tests pass No more tests
Choose a test
Start
Compilation error
Link error
DONE!
Programming error
Compiles Clean
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Make the Test Compile
22
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Make the Test Compile
23
#ifndef D_CircularBuffer_H#define D_CircularBuffer_H
typedef struct CircularBuffer CircularBuffer;
CircularBuffer * CircularBuffer_Create();void CircularBuffer_Destroy(CircularBuffer *);int CircularBuffer_IsEmpty(CircularBuffer *);
#endif // D_CircularBuffer_H
Write the test
Make the test link
Make the test compile
Compilation error
Link error
New test failsMake the test pass
Refactor(Make it right)
All tests pass
All tests pass No more tests
Choose a test
Start
Compilation error
Link error
DONE!
Programming error
Compiles Clean
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Make the Test Link
24
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Make the Test Linkand Intentionally Fail
25
#include "CircularBuffer.h"#include <stdlib.h>
struct CircularBuffer{ int dummy;} ;
... not all code shown ...
int CircularBuffer_IsEmpty(CircularBuffer* self){ return 0;}
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Watch it Fail
26
CircularBufferTest.cpp:48: error: Failure in TEST(CircularBuffer, ShouldBeEmptyAfterCreate)
CHECK(CircularBuffer_IsEmpty(buffer)) failed
Write the test
Make the test link
Make the test compile
Compilation error
Link error
New test failsMake the test pass
Refactor(Make it right)
All tests pass
All tests pass No more tests
Choose a test
Start
Compilation error
Link error
DONE!
Programming error
Compiles Clean
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Make the Test Pass
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Write no More Code than is Necessary to Pass the Current Test
• Hard code behavior or partial implementations are OK and often preferred.
28
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Make the Test Pass
29
#include "CircularBuffer.h"#include <stdlib.h>
struct CircularBuffer{ int dummy;} ;
... not all code shown ...
int CircularBuffer_IsEmpty(CircularBuffer* self){ return 1;}
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Repeat until You Run Out of Tests
• The first tests drive the interface definition• Later tests complete the behavior• You will think of more tests as you go
– Write the new test– or Add it to the test list
30
Code without
new feature
Code withnew tested
feature
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Exercise
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Post-Exercise Discussion
32
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
But that is so Inefficient!So Slow!
Show us the fast way!
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Get good at being careful and you can go fast.
34
The Careful way is the Fast Way!
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Why is it difficult to sustain?
35
TDD:• Steady pace• Speed limits• No traffic lights (bugs)• Might feel slow!!
Traditional:• Spurts• Fast when no problems!• Debugging• Feels fast! But often slow
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Sustainability
36
Speculate Code Test DebugTraditional development
Time developers do not notice nor
plan for
Time vs
Test-driven development
Speculate Test and Code Debug
Feels slower
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
37
Stuff you need to know!
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Tests Must Be Automated
38
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
39
It is not a unit test when:• It talks to the database• It communicates across the network• It touches the file system• It can't run at the same time
as any of your other unit tests• You have to do special things to your
environment (such as editing config files) to run it.
Michael Featherswww.renaissancesoftware.net
[email protected] © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Refactoring
40
Structured code transformation to prepare the code
for change
Key points:• Doesn’t adjust functionality• Small and disciplined• Well defined• Keeps code healthy
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Code Smells
4133
A sign there is probably something wrong with your code
It smells!
It stinks!
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Modularity == Testability
42
Focus on tests first ensures modularity and other good design principles
It’s about design, stupid!
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
43
Design from the perspective of userather than implementation
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
CppUTest Cheat Sheet
44
/* in CheatSheetTest.cpp */#include "CppUTest/TestHarness.h"
/* Declare TestGroup with name CheatSheet */TEST_GROUP(CheatSheet) {/* declare a setup method for the test group. Optional. */ void setup () {/* Set method real_one to stub. Automatically restore in teardown */ UT_PTR_SET(real_one, stub); }
/* Declare a teardown method for the test group. Optional */ void teardown() { }}; /* Do not forget semicolumn */
/* Declare one test within the test group */TEST(CheatSheet, TestName){ /* Check two longs are equal */ LONGS_EQUAL(1, 1);
/* Check a condition */ CHECK(true == true);
/* Check a string */ STRCMP_EQUAL("HelloWorld", "HelloWorld");}
/* In allTest.cpp */IMPORT_TEST_GROUP(CheatSheet);
/* In main.cpp */
#include "CppUTest/CommandLineTestRunner.h"#include "AllTests.h"
int main(int ac, char** av){ return CommandLineTestRunner::RunAllTests(ac, av);}
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
CppUTest Mocking Cheat Sheet
45
/* Additional include from CppUTestExt */#include "CppUTest/TestHarness.h"#include "CppUTestExt/MockSupport.h"
TEST_GROUP(MockCheatSheet){ void teardown() { /* Check expectations. Alternatively use MockSupportPlugin */ mock().checkExpectations(); }};
TEST(MockCheatSheet, foo){ /* Record 2 calls to Foo. Return different values on each call */ mock().expectOneCall("Foo") .withParameter("param_string", "value_string") .withParameter("param_int", 10) .andReturnValue(30); mock().expectOneCall("Foo") .ignoreOtherParameters() .andReturnValue(50);
/* Call production code */ productionCodeFooCalls();}
TEST(MockCheatSheet, bar){ /* Expect 2 calls on Bar. Check only one parameter */ mock().expectNCalls(2, "Bar") .withParameter("param_double", 1.5) .ignoreOtherParameters();
/* And the production code call */ productionCodeBarCalls();}
/* Stubbed out product code using linker, function pointer, or overriding */int foo(const char* param_string, int param_int){ /* Tell CppUTest Mocking what we mock. Also return recorded value */ return mock().actualCall("Foo") .withParameter("param_string", param_string) .withParameter("param_int", param_int) .returnValue().getIntValue();}
void bar(double param_double, const char* param_string){ mock().actualCall("Bar") .withParameter("param_double", param_double) .withParameter("param_string", param_string);}
/* Production code calls to the methods we stubbed */void productionCodeFooCalls(){ int return_value; return_value = foo("value_string", 10); return_value = foo("value_string", 10);}
void productionCodeBarCalls(){ bar(1.5, "more"); bar(1.5, "more");}
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Embedded
46
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Use C or C++?
• Why C++ (e.g. gcc):
• Able to use C++ unit test framework
• Able to use C++ features in tests
• Why C:
• Not annoyed by the small differences
• Able to use a C compiler.
• E.g. run tests in “real environment”
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Hardware is Scarce!
• It does not exist.• It is being used by
someone else.• It has bugs of its
own.
48
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
The Real Thing?
• Run unit tests on real hardware?
• Probably not. Too slow.
• Every now and then it could be possible.
• Use the real compiler?
• Periodically. With check in.
• Do run higher level tests on real HW every now and then!
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
But There are Risks with Development System Tests
• Architecture differences– Word size– Big-endian Little-endian– Alignment
• Compiler differences• Library differences• Execution differences
50
Write a TestMake it Pass
Refactor
Stage 1
Compile for TargetProcessor
Stage 2
Run Tests in the Eval Hardware
Stage 3
Run Testsin Target Hardware
Stage 4
AcceptanceTests
Stage 5
More Frequent Less Frequent
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
TDD Adaptation for Embedded Development
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
TDD Adaptation for Embedded Development
52
See : http://renaissancesoftware.net/files/articles/ProgressBeforeHardware.pdf
Write a TestMake it Pass
Refactor
Stage 1
Compile for TargetProcessor
Stage 2
Run Tests in the Eval Hardware
Stage 3
Run Testsin Target Hardware
Stage 4
AcceptanceTests
Stage 5
More Frequent Less Frequent
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Problems Found at Appropriate Stage
53
Stage Problems Likely to Find
1 Logic, design, modularity, interface, boundary conditions
2 Compiler compatibility (language features)Library compatibility (header files, prototypes)
3 Processor executions problems (bugs in compiler and standard libraries)Portability problems (word size, alignment, endian)
4 Ditto stage 3Hardware integration problemsMisunderstood hardware specifications
5 Ditto stage 4Misunderstood feature specification
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
How do I test Code with Hardware and OS Dependencies?
54
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
A Module with Collaborators
• Every minute, the RTOS wakes up the Light Scheduler.
• If it is time for one of the lights to be controlled, the LightController is told to turn on/off the light.
55
Time Service
+ GetTime()+ SetPeriodicAlarm()
LightScheduler
+ScheduleTurnOn()+RemoveSchedule()+WakeUp()
Light Controller
+ On(id)+ Off(id)
Hardware RTOS
<<anonymous callback>>
Admin Console
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Program to Interfaces
• Separate interface and implementation as separate entities.
• This design has good separation of responsibilities
56
<<interface>>Time Service
+ GetTime()+ SetPeriodicAlarm()
LightScheduler
+ ScheduleTurnOn()+ RemoveSchedule()+WakeUp()
<<interface>>Light Controller
+ On(id)+ Off(id)
Model 42 Hardware RTOS
<<anonymous callback>>
Model 42Light Controller
RTOS Time Service
<<implements>> <<implements>>
Admin Console
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Testing a Module with Collaborators
• Use the real collaborators if you can.
• Use fakes when you must.
57
<<interface>>Time Service
+ GetTime()+ SetPeriodicAlarm()
LightScheduler
Test
LightScheduler
+ ScheduleTurnOn()+ RemoveSchedule()+wakeUp()
<<interface>>Light Controller
+ On(id)+ Off(id)
Light Controller Spy
FakeTime Service
<<implements>> <<implements>>
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Testing the Scheduler
58
TEST(LightScheduler, ScheduleOnTodayNotTimeYet){ LightScheduler_ScheduleTurnOn(3, EVERYDAY, 1000); FakeTimeSource_SetMinute(999);
LightScheduler_Wakeup();
LONGS_EQUAL(LIGHT_NA, LightControllerSpy_GetState(3));}
TEST(LightScheduler, ScheduleOnTodayItsTime){ LightScheduler_ScheduleTurnOn(3, EVERYDAY, 1000); FakeTimeSource_SetMinute(1000);
LightScheduler_Wakeup();
LONGS_EQUAL(LIGHT_ON, LightControllerSpy_GetState(3));}
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. TDD with Collaborating Objects
Partial Skeleton Let’s You Try the Design and Test Ideas Early
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. TDD with Collaborating Objects
f2fp refactoring
void function (int para); void function (int para){ do_implementation ();}
Header Source
before
afterextern void (*function) (int para);
void function_imp (int para){ do_implementation ();}
void (*function)(int para) = function_imp;
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Stubbing
• Stub level:
• Preprocessor (rare)
• Function pointer
• Link
• Stub type
• Exploding
• Sensible default
• Recording
• Generic
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Level: preprocessor
#include "stubs.h"
void something (){ function(100);}
#define function(a) function_stub(a, b)
Source Stub header
Advantages: Disadvantages:
- Creates lots of flexibility.
-> for example. Can stub out just one call.
- Changes the production code
- Requires different build configurations
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Level: function pointer
Advantages: Disadvantages:
- Ability to runtime change
- Pretty safe
- Requires f2fp refactoring
- One extra call
- Dangling function points (avoided when using UT_PTR_SET)
TEST_GROUP(group){ void setup () { UT_PTR_SET(real_one, stub); }
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Level: link stubs
Advantages: Disadvantages:
- Can give a lot of flexibility!
- No change in production code
- Easy to re-use stubs
- Be careful of different configurations. Just have one stub (use generic link stubs)
- Difficult (impossible) to call “the real thing”’
- Don’t use on file-level. Otherwise it causes many executables and stubs.
void function (){ /* Do stubbed things */}
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Type: Sensible default
int function (){ return 10;}
Useful when stubbing out 3rd party libraries
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Type: Exploding
void function (){ FAIL("stub: function was called");}
Especially useful when starting. When it explodes something went wrong or you need to implement it.
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Type: Dynamic
Benefits of both function pointer and link level stubs!
void (*function_stub) () = NULL;
void function (){ if (function_stub) function_stub ();}
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Type: Recording (mock)
Very generic usage. int function (int parameter){ mock().actualCall("function") .withParameter("parameter", parameter) .returnValue().getIntValue();}
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Type: Combination
int function (int parameter){ if (function_stub) return function_stub(parameter); mock().actualCall("function") .withParameter("parameter", parameter);
if (mock().hasReturnValue()) return mock().returnValue().getIntValue(); return 0;}
Very flexible.
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Closing Thoughts
70
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Why is it Worth Changing How You Program
• Defects kill predictability:–Cost of fixing is not predictable–When they materialize is not predictable
• Test-driven is predictable:–Working at a steady pace–Results in fewer bugs–More productive than “debug-later programming”
• Test-driven programmers rarely need the debugger
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
TDD is Defect Prevention
72
Test and FixTest and Fix
Test and Fix
Test and Fix
Test and FixTest and Fix
Test and Fix
Test and Fix
Test and FixTest and Fix
Test and FixTest and Fix
Test and Fix
Test and FixTest and Fix
CAN PREVENT CODE FIRES
Test and Fix
Test and Fix
Test and Fix
Copyright © 2008-2011 James W. Grenning, Bas VoddeAll Rights Reserved. .
Questions
73