Upload
agiletourchina
View
1.263
Download
1
Embed Size (px)
DESCRIPTION
Citation preview
Copyright ©2008-2009 James W. GrenningAll Rights Reserved. . Photos from James Grenning’s Personal Collection or Wiki-Commons
Test is NOT About Finding Bugs!
James W GrenningRenaissance Software Consulting, Co.
[email protected]: jwgrenning
1
1Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
2
2Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
www.agilemanifesto.org
3
3Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
4
4Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
More Functionality in Software
6 years
5
5Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
More and More Software
6
6Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
7
7Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
8
8Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
9
9Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
10
10Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
11
11Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
12
12Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
13
13Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
14
14Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
15
15Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
16
16Monday, October 18, 2010
2000’s
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Now vs. Then
17
SW
SW
HW
1960’s
SW
HW
17Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
The Nature of Software
• Software is fragile– The nature of discrete systems
• Any change can break just about anything• Test and forget model leads to big surprises and
problems
18
18Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
19
19Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Lot’s of Software meansLot’s of Problems
• 10-50 defects per 1000 Lines of Code (KLOC)
• How many bugs in
20
20Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
21
21Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Voyager
22
22Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
23
23Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Mars Climate (burns in) Orbiter
24
24Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
25
25Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
1990 - AT&T #4ESS
26
26Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
27
27Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
28
28Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
29
Did you say something?
What the heck!?! Where are we?
29Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
30
30Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Defect Found During Simulation
31
31Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
32
32Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
33
33Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
34
34Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Don’t Test to Find DefectsTest to Prevent Defects
• Automate• Test as you go• Re-test everything
• Code is guilty until proven innocent
35
35Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Manual Test Is Unsustainable
36
36Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Et = f(Ed)Et is the effort to test a new feature, and is a function of the effort to develop the feature.
Ed is the effort to develop a new featureAssume a constant linear relationship
Et = f(Ed) = KEd
37
37Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Conservative Estimate of Developer to Tester Ratio
38
0
5
10
15
1 2 3 4 5 6 7 8 9 10
Num
ber
of P
eopl
e
Iteration
New functionality manual test effort
New functionality development effort
38Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
It’s not that EasySoftware is Fragile
• 25% of defects are introduced while changing existing code (study by Grady)– The new code works– Some existing code is unintentionally broken
39
39Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Etn = f(Ed) + C Et(n)Etn is the effort to fully test a product at iteration N
Because any change can break previously working software, we must retest.
Etn is a function of the effort to develop the feature plus some fraction of the effort to test all previous iterations.
40
i = 0
n
x
40Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Unsustainable Growth with any Coefficient
41
0
5
10
15
1 2 3 4 5 6 7 8 9 10
Peop
le N
eede
d
Iteration
Assume regression test effort is 50% original manual test effort
Total manual test effort is unsustainable
Ever Growing Need
Flat Budget
Unt
este
d C
ode
Gap
41Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
So, Maybe we should only test at the end of the development cycle
42
42Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
FBI Case-File System
43
43Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Cost of BDUF gone bad
44
44Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Should We Wait Until the End to Test?
Requirements
Code
Design
Test
Time
NovemberSeptemberJulyMayMarch
Test and Fix
Test and Fix
Test and Fix
Test and FixTest and Fix
Test and Fix
Test and Fix
Test and Fix
Test and Fix
Test and FixTest and Fix
Test and Fix
Test and Fix
Test and FixTest and Fix
Test and FixTest and Fix
45
Test
45Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
This Work Flow is Designed to Allow Defects
46
Development
Test
Defects
46Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Development and Test Work TogetherPreventing Defects
47
Development
Test
47Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Testing is not a Phase
• Testing starts on day one• Tests provide the
specification of what is to be developed
• QA/System Test moves upstream.
48
HELP!
48Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Edsger Dijkstra
Those who want really reliable software will discover that they must find means of avoiding the majority of bugs to start with, and as a result, the programming process will become cheaper. If you want more effective programmers, you will discover that they should not waste their time debugging, they should not introduce the bugs to start with.
49
49Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Can we Realize Dijkstra’s Dream andPrevent Defects with
Test Driven Development?
50
50Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Zune 30G
• On December 31, 2008 this function went into an infinite loop.
• Ruining New Years Eve Parties around the world.
51
51Monday, October 18, 2010
The buggy function:
It accepts the number of days since January 1, 1980 and figures out: day, date month and year.
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
52
Zune 30G Bug
BOOL ConvertDays(UINT32 days, SYSTEMTIME* lpTime)
52Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
53
This Test Could Have Prevented the Zune Bug
TEST(Rtc, check20081231){ days = daysSince1980(2008, 366); CHECK(ConvertDays(days, &time)); assertDate(WED, 31, 12, 2008);}
53Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
What Tests do we Need?
• One Test Would Have Prevented Zune Bug• If we knew where the bugs are, we could just write
those tests (or fix the bug).• Bugs can be anywhere, so we have to write tests for
everything.
54
54Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Uncle Bob’s Three Rules of TDD
1.You are not allowed to write any production code unless it is to make a failing unit test pass.
2.You are not allowed to write any more of a unit test than is sufficient to fail; and compilation failures are failures.
3.You are not allowed to write any more production code than is sufficient to pass the one failing unit test.
source: Robert C. Martin, blog.objectmentor.com
55
55Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
TDD State Machine
56
Write the test
Deal with the linker
Deal with the Compiler
Compilation error
Link error
New test failsMake it work
Refactor(Make it right)
All tests pass
All tests pass No more tests
Choose a test
Start
Compilation error
Link error
DONE!
Programming error
Compiles Clean
56Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
The Physics of Debug Later Programming (DLP)
• As Td increases, Tfind increases dramatically• Tfix is usually short, but can increase with Td
57
Bug discoveryMistake made(bug injection)
Bug found Bug fixed
Td Tfind T fix
Time
57Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
The Physics of Test Driven Development
• When Td approaches zero, Tfind approaches zero• In many cases, bugs are not around long enough to be considered
bugs.• See: http://www.renaissancesoftware.net/blog/archives/16
58
Mistake discovery
Mistake made
Root cause found
Mistake fixed
T d Tfind T fix
Time
58Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
TDD Progress in the Right Direction
• See: http://www.renaissancesoftware.net/blog/archives/33
59
Code
without
new feature
Code with
new untested
feature
Code with
new tested
feature
Write all production code at once
Debug
Test Driven Development
Write the test
Deal with the linker
Deal with the Compiler
Compilation error
Link error
New test failsMake it work
Refactor(Make it right)
All tests pass
All tests pass No more tests
Choose a test
Start
Compilation error
Link error
DONE!
Programming error
Compiles Clean
59Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
How Can we Keep the Cost or Retest Sustainable? -- Automate
60
0
5
10
15
1 2 3 4 5 6 7 8 9 10
Peop
le N
eede
d
Iteration
Assume automation test effort is 100% more than original manual test effort
Relatively Flat and Sustainable
60Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
How Can we Keep the Cost or Retest Sustainable? -- Automate
61
0
5
10
15
1 2 3 4 5 6 7 8 9 10
Peop
le N
eede
d
Iteration
Relatively Flat and Sustainable
Ever Growing Need
Flat Budget
Unt
este
d C
ode
Gap
61Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
But TDD won’t work for us!We have special problems!
• Fire alarm• Home automation• Banking• Insurance• Robotics• Medical electronics• Printing• Gambling machines
• Telecom• Police radio• Cell phone• On-line gaming• Security software• Mesh networks• Test equipment• Medical records
62
62Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Yes, You are Special and Unique
• But TDD can work in many environments
63
63Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
But I’ve seen it work.Everywhere someone lets it.
• Fire alarm• Home automation• Banking• Insurance• Robotics• Medical electronics• Printing• Gambling machines
• Telecom• Police radio• Cell phone• On-line gaming• Security software• Mesh networks• Test equipment• Medical records
64
64Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
But I’m special
I have ____________ Dependencies
65
Fill in the blank*
*Hardware, OS, Database, Regulatory, Real-time, User, Network...
65Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Unmanaged Hardware Dependency Lead to Manual Testing
66
The Net
Core Software
66Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
But I Have Hardware Dependencies
• Break the dependencies
67
<<interface>>
Time Service
+ getDay()+ getTimeOfDay()+ setPeriodicAlarm()
LightScheduler
Test
LightScheduler
+addSchedule()+removeSchedule()+wakeUp()
<<interface>>
Light Controller
+ on(id)+ off(id)
Light Controller Spy
FakeTime Service
<<implements>> <<implements>>
67Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Design Must Accommodate Test
TEST(LightScheduler, TurnOnEverydayNotTimeYet) { LightScheduler_TurnOn(3, TimeService::EVERYDAY, 510); whenItBecomes(TimeService::MONDAY, 509); expectLight(3, LIGHT_UNCHANGED); }
TEST(LightScheduler, TurnOnEveryday) { LightScheduler_TurnOn(3, TimeService::EVERYDAY, 510); whenItBecomes(TimeService::MONDAY, 510); expectLight(3, LIGHT_ON); }
68
68Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Managed Dependencies Enables Automated Testing
69
Copyright (c) 2009 James Grenning
Core Software
Test scripts
Test Agent
Test Controller
69Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
There are Different kinds of Automated Tests
• Unit Test – Feedback to the developer that the code does what is expected– Written using a Unit Test Harness (e.g. unity, CppUTest, …)
• Story Tests - Executable use cases - Our focus– Feedback to the Customer that the code meets the requirements– Used at many levels
– Component– Groups of integrated components– System
– Written in a domain specific language (e.g. FitNesse) [FITNESSE]
• Integration, System, Load Tests...
70
70Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Unit tests are critical
• A house made from solid bricks can stand a long time• TDD gives solid building blocks• Unit tests tell programmers that code works like they
think it works
71
71Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
System Level Tests Cannot be Thorough
• 1000 (or more) tests are needed to test this simple system
72
10 states
10 states
10 states
5 interactions
5 interactions
5 interactions
Tests
72Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
TDD unit tests can be thorough
• As few as 30 unit tests and 15 integration test when tested as units
73
10 states
10 states
10 states
5 interactions
5 interactions
5 interactions
Tests Tests
73Monday, October 18, 2010
Copyright ©2008-2009 James W. GrenningAll Rights Reserved. . Photos from James Grenning’s Personal Collection or Wiki-Commons
Higher Level Tests are Needed Too
Could Test Help Specify what is needed with Executable Use cases
See http://renaissancesoftware.net/papers.htmlFor Executable Use Cases Paper
74
74Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Use-Case Template
• Name:• Goal: • Preconditions:• Success End Condition: • Failed End Condition: • Primary Actor:• Trigger:• MAIN SUCCESS SCENARIO• EXTENSIONS
Source Alistair Cockburn http://alistair.cockburn.us/Basic+use+case+template
75
75Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Use-Case Example
76
Information Description
Name Schedule light control Goal Allow system users to schedule lights to turn on, off, or
dimPreconditions System has controllable lights attachedSuccess End Condition The scheduled light has been controlled at the scheduled
timeFailed End Condition The scheduled light has not been controlled at the
scheduled timePrimary Actor Home ownerTrigger Scheduled time is reachedMain Success Scenario 1.The home owner schedules a light to turn on at a
specific time on a specific day
2.The scheduler wakes up at the right time of the right day
3.The light scheduled for this minute is turned on
76Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Use-Case ExampleContinued
77
Information Description
Extensions/Variations 1a. Homeowner can schedule the light to turn on
1b. Homeowner can schedule the light to turn off
1c. Homeowner can schedule the light to set to a dim
level
1d. Homeowner can specify weekend schedule
1e. Homeowner can specify weekday schedule
2a - Scheduler does nothing when it wakes up and there
are no scheduled controls.
3a - Light is turned on when on is scheduled
3b - Light is turned off when off is scheduled
3c - Light is set to a specified level when dim is scheduled
77Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Executable Use-Case with FitNesse
78
Items in tables are tell to FitNesse how to interact with the system under test
Initialization
Cleanup
78Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Passing Test
79
79Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Failing Test
80
80Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
A Suite of Tests While all Test Pass
81
81Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
When the Story Test is Ready Before the Development Finishes
82
82Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
When Things are not Working
83
83Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
FitNesse Test Architecture
84
<<interface>>
Time Service
+ GetTime()+ SetPeriodicAlarm()
LightScheduler
Fixture
LightScheduler
+ ScheduleTurnOn()+ RemoveSchedule()+wakeUp()
<<interface>>
Light Controller
+ On(id)+ Off(id)
Light Controller Spy
FakeTime Service
<<implements>> <<implements>>
FitNesseWiki Server
CSlim Library
<<socket>>FitNesse Test Pages
84Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Test is NOT About Finding Bugs!
• TDD to prevent bugs• Automate repetitious tests
– to keep bugs out– to keep the cost of re-test low
• It’s not easy, until you learn how to do it– then it’s still not always easy, but worth it
85
85Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Don’t let your Product Appear on a Future Famous Failures Presentation
86
86Monday, October 18, 2010
Copyright ©2008-2009 James W. GrenningAll Rights Reserved. . Photos from James Grenning’s Personal Collection or Wiki-Commons
Thank You!
87
87Monday, October 18, 2010
Copyright ©2008-2010 James W. GrenningAll Rights Reserved.
Available now in Beta
http://pragprog.com/titles/jgade/
In print early 2011
88
88Monday, October 18, 2010