Agile Developers Go TestingPart II

Darcy Casselman @flying_squirrelAlexei Zheglov @az1

November 2010

What are we talking about today?

• A lot of things…• They may seem disconnected, but they all:

– improve software quality– involve testing– done by developers

• None of them can do the job alone• We cannot:

– Talk about all of them– Give them equal time

Unit Testing

• We showed you a very simple unit test example earlier

• We’ll show you a more difficult example later• Let’s step back first

Focus on Quality

quotes

"...in 2000, during the dot-com bubble, ... software quality for North American teams... ranged from 6 defects per function point to less than 3 per 100 function point, a range of 200 to 1.  The midpoint is approximately 1 defect per 0.6 to 1.0 function points.  This implies that it is common for teams to spend more than 90 percent of their effort fixing defects.”

http://www.amazon.com/Kanban-David-J-Anderson/dp/0984521402/http://www.amazon.com/Software-Assessments-Benchmarks-Best-Practices/dp/0201485427/

Logical Connection?

“The midpoint is approximately 1 defect per 0.6 to 1.0 function points.  This implies that it is common for teams to spend more than 90 percent of their effort fixing defects.”

How does he know what they are spending their time on?

Capacity Allocation To Fixing Defects

=

Capacity Allocation To Fixing Defects

Defect Injection Rate

Bugs, Features

http://www.flickr.com/photos/dratz/1045336659/

Another View

• Root-cause analysis of established practices

• E.g. unit testing• Five-whys applied to

the solution, not the problem

• The link between the third and fourth whys?

Source: Karl Scotland http://availagility.co.uk/2010/10/04/a-root-cause-analysis-of-agile-practices/

Compare Two Teams

Team A• Just finished 40 “features”

(units of work)• Introduced 10 “bugs” (units

of rework)

Team B• 10 features• 8 bugs

Team A Progression

• Terms beyond the first three are negligible• Most defects eliminated in the next 2 releases• The 2nd maintenance release scope is very small• Illusion that any team can do it• Time spent fixing defects: 25%

3153...5.21040

Team B Progression

• The sum of the first 3 terms: only 24.4 (less than half of the total!)

• The sum of the first 10: approx. 45• Time spent fixing defects: 80%

50...12.54.6810

We want to be more like Team A

and certainly not like Team B

(or better)

Testing

• Is what we are talking about today• Leaving out other technical practices• Remembering that testing itself does not

provide quality

Test MatrixWhy?

Unit Did we build the code right? Does it work as specified?

Test MatrixWhy?

Unit Did we build the code right? Does it work as specified?

Acceptance Did we build the right code? Does it do what the user wants?

Test MatrixWhy?

Unit Did we build the code right? Does it work as specified?

Acceptance Did we build the right code? Does it do what the user wants?

Integration Do system components work together? What about performance?

Test MatrixWhy?

Unit Did we build the code right? Does it work as specified?

Acceptance Did we build the right code? Does it do what the user wants?

Integration Do system components work together? What about performance?

System Is the system installed and configured correctly?

Test MatrixWhy?

Unit Did we build the code right? Does it work as specified?

Acceptance Did we build the right code? Does it do what the user wants?

Integration Do system components work together? What about performance?

System Is the system installed and configured correctly?

Exploratory What can we find out about the system that is not specified by these tests?

Test MatrixCoverage

%Auto

?Tools Developers’ Role

Unit

Acceptance

Integration

System

Exploratory

Test Matrix

Sources: Robert C. Martin, Rob Diana http://regulargeek.com/2009/06/12/how-much-testing-is-enough/

Coverage %

Auto?

Tools Developers’ Role

Unit Yes NUnit/JUnit and similar toolsisolation (mocking) frameworks

Acceptance Yes acceptance testing frameworks (Cucumber, FitNesse, Robot etc.)domain-specific languages (DSLs)

Integration Yes UI testing frameworks (e.g. Selenium)

System Yes same

Exploratory No -

Test Matrix

Sources: Robert C. Martin, Rob Diana http://regulargeek.com/2009/06/12/how-much-testing-is-enough/

Coverage %

Auto?

Tools Developers’ Role

Unit 90% Yes NUnit/JUnit and similar toolsisolation (mocking) frameworks

Acceptance 50% Yes acceptance testing frameworks (Cucumber, FitNesse, Robot etc.)domain-specific languages (DSLs)

Integration 20% Yes UI testing frameworks (e.g. Selenium)

System 5% Yes same

Exploratory - No -

Test Matrix

Sources: Robert C. Martin, Rob Diana http://regulargeek.com/2009/06/12/how-much-testing-is-enough/

Coverage %

Auto?

Tools Developers’ Role

Unit 90% Yes NUnit/JUnit and similar toolsisolation (mocking) frameworks

take personal responsibility

Acceptance 50% Yes acceptance testing frameworks (Cucumber, FitNesse, Robot etc.)domain-specific languages (DSLs)

collaborateIntegration 20% Yes UI testing frameworks (e.g. Selenium)

System 5% Yes same

Exploratory - No - -

A Very Short Introduction to Acceptance Testing

Consider This Test Script

• A new customer registers• The customer places three books in the

shopping cart• The customer goes to check-out• The customer fills in their delivery address

(province: Ontario)• The system offers free delivery to the customer

Source: Gojko Adzic. Bridging the Communication Gap, pp. 78-79http://www.amazon.com/gp/product/0955683610

Compare the Test Script to This

The system offers free delivery to Ontario customers when they place their first order and their order contains at least three books

Convert to the Tabular Form

number of books order number ship to free delivery?

3 1 Ontario Yes

number of books order number ship to free delivery?

3 1 Ontario Yes

2 1 Ontario No

number of books order number ship to free delivery?

3 1 Ontario Yes

2 1 Ontario No

3 2 Ontario No

number of books order number ship to free delivery?

3 1 Ontario Yes

2 1 Ontario No

3 2 Ontario No

3 1 Alberta No

How To Do It

• Collaborate:– developers– business analysts– testers– product owner

• Which tool to use is not as important• Don’t look up terminology in Wikipedia

Books

http://www.amazon.com/gp/product/0955683610http://manning.com/adzic/

Unit-TestingWhen System Under Test (SUT)

Has Dependencies

Break the dependencies

What If We Don’t?

• Configuration• Database data• Remote services• Permissions• Time to run• Infrequent errors• Dependencies in development

Can You Run Unit Tests Like This?

http://www.flickr.com/photos/theredproject/2878006012/

Code Example

NotificationService

StudentRepositoryCaller

GradingEngine

Code Example

Mock

StubUnit Test

System Under Test

(Open IDE)

37.62%

34.42%

9.01%

4.05%

3.12%

1.99%

1.77%

1.21%

0.92%

0.57%

3.97% 1.35%

Popular Frameworks (.NET)

MoqRhinoMocksHand-rolledTypemock IsolatorMolesJustMockNSubstituteNMock2NMockFakeItEasyNoneOther

Source http://www.osherove.com/blog/2010/9/10/2010-poll-which-isolation-framework-do-you-use-in-net.html September 2010

Two Camps

Design for Testability:Moq, RhinoMocks, etc.

• Dynamically fake objects implementing interfaces/base classes in their own way

• Require software under test to be designed to be testable

• Programmers to follow SOLID principles (esp. Dependency Inversion Principle)

Mock Anything, e.g.Typemock Isolator, JustMock

• Rely on Profiler API• Can intercept anything and

run something else instead• Can fake statics, sealed

classes, etc.• Don’t force design for

testability

Popular Frameworks (Java)

• “Conventional”:– Mockito http://code.google.com/p/mockito/– EasyMock http://easymock.org/– JMock http://www.jmock.org/

• Alternative:– JMockit http://code.google.com/p/jmockit/

Roy Osherove

• The Chief Architect at TypeMock• Wrote: “A customer switches to Moq – and

I’m happy”• And: “This is pure awesome. This is why I

come to work every day.”• How is it possible?• Read it:

http://www.osherove.com/blog/2010/9/29/a-customer-switches-to-moq-and-im-happy.html

Back to Our Code Example

• Grading engine operates on Student and Grade

• But Student has a Department• And Grade has a Course and Course has a

Semester• We start mocking• Soon, we have a lot of mocks

“Every time a mock returns another mock a fairy dies”

• Who said it?• Not the guy whose tweet comes up first in

Google search results• I am not 100% sure• What the heck does it mean anyway?

“Every time a mock returns another mock a fairy dies”

• Who said it?• Not the guy whose tweet comes up first in

Google search results• I am not 100% sure• What the heck does it mean anyway?• We don’t have time for it in this presentation

http://www.amazon.com/Growing-Object-Oriented-Software-Guided-Tests/dp/0321503627/

Creating Objects

• Constructor injection (as in our code example)• Factory• What if dependencies have dependencies?• What if dependencies of dependencies have

dependencies?

Pretty Soon, Things Begin To Look Like This

Source: http://ayende.com/Blog/archive/2010/01/22/rejecting-dependency-injection-inversion.aspx

(different box color = different life span)

(and this is a simple example)

These Tools Can Help

• StructureMap• Castle Windsor• Unity• NInject• autofac

Conclusions

Quality: More Important Than This

http://www.flickr.com/photos/markhillary/2681349855/

More Conclusions

• You have to test• No single testing strategy is nearly enough• Employ multiple strategies• Each of them is difficult to master• We barely scratched the surface today• Agile brings testing into focus• Combine hard and soft skills• Collaborate

Q&A

http://creativecommons.org/licenses/by-nd/3.0/