Agile Testing Workshop - itq.chitq.ch/pdf/sqm/07/ics_jorgensen_agile_testing.pdf · Agile Testing Workshop SQC 2007, Düsseldorf Paul C. Jorgensen, Ph.D. The Agile Manifesto “We

Agile Testing Workshop SQC 2007, Düsseldorf Paul C. Jorgensen, Ph.D.

Agile Testing Workshop

Paul C. Jorgensen, Ph.D., ProfessorSchool of Computing and Information Systems

Grand Valley State UniversityAllendale, Michigan 49491


Overview/Übersicht

• The Agile Development Paradigm • Flavors of Agile Testing

– Context-Driven Testing – Good Enough Testing – Exploratory Testing – Test-Driven Development (TDD)

• Comparison of MDD and TDD • Open Questions and Conclusions


The Agile Development Paradigm

• the Agile Manifesto • more than just “Something for lazy

Americans...”• Customer driven (via User Stories) • Bottom-up rather than traditional

Top-down • Agile or Fragile?


The Agile Manifesto“We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and toolsWorking software over comprehensive

documentationCustomer collaboration over contract negotiationResponding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more. “

(signed by 17 people; http://agilemanifesto.org/)


Principles behind the Agile Manifesto

• “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

• “Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.

• “Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

• “Business people and developers must work together daily throughout the project.


Principles behind the Agile Manifesto(continued)

• “Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

• “The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

• “Working software is the primary measure of progress.

• “Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.


Principles behind the Agile Manifesto(concluded)

• “Continuous attention to technical excellence and good design enhances agility.

• “Simplicity--the art of maximizing the amount of work not done--is essential.

• “The best architectures, requirements, and designs emerge from self-organizing teams.

• “At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.”


BUT—what Happened to Testing?

• Mike Beedle (one of the group that signed the Manifesto) suggested in 2004 that another “value”should be added:

• we have come to value...

Tests over requirements definition and traceability

(Brian Marick’s Blog, http://www.testing.com/cgi-bin/blog/2004/03/16)


Development with User Stories

User Stories

Test CasesIncrement Plan

Coding

Unit Test

Increment

System Test Small Release


Nun singen wir, alle zusammen(Melodie von Josef Haydn, 1797)

Agile, agile, über alles, Über alles in der Welt.Agile, agile, über alles, Über alles in der Welt.Kein mehr Wasserfall, kein mehr Prozess,Kein mehr Requirements und Design.Agile, agile, über alles, Über alles in der Welt.


Testing in an Agile Development

• Context-Driven Testing• Good Enough Testing• Exploratory Testing• Test-Driven Development


Context-Driven Testing(seven basic principles)

1. “The value of any practice depends on its context.

2. There are good practices in context, but there are no best practices.

3. People, working together, are the most important part of any project's context.

4. Projects unfold over time in ways that are often not predictable.

5. The product is a solution. If the problem isn't solved, the product doesn't work.

6. Good software testing is a challenging intellectual process.

7. Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.”


Observations and Conclusions• Principles 1 and 2 apply to almost any endeavor.• Principle 3 applies to all forms of teamwork.• Agile development is a good response to Principle 4 • Principles 5, 6, and 7 apply to software testing.• These principles are so general, they give little

direction about HOW to do context-driven testing.• Context-driven testing depends primarily on:

– Skill– knowledge– Ingenuity– Motivation

• Craftsmanship!


Good Enough Testing“I want to distinguish

the Good Enough approach from the approach that says that we should keep testing until Management pries the product from my cold, dead fingers.” --James Bach

“Good Enough testing is the process of developing a sufficient assessment of quality, at a reasonable cost, to enable wise and timely decisions to be made concerning the product.”


Bach’s Principles of Good Enough Testing

• Assess product quality– How accurate/complete is the testing?

• Evaluate the cost of testing– Within budget?– Any redundancy? Unnecessary testing?

• Determine whether testing supports necessary decision making– Risk?

• Are testing results timely?– Point of diminishing returns.


Observations and Conclusions

• Good Enough Testing can be applied to any software development. It is not necessarily Agile.

• Very sensible, but it requires effort from both testers and management.

• No guidelines—just generalities.


Exploratory Testing

• Explore: to investigate the unknown • “The opposite of scripted testing”—James Bach• More than ad hoc testing• Shares some pros and cons with Special Value Testing• Depends on the abilities of the exploratory tester

– Curiosity– Ingenuity– Time and tools

• A discovery process• A learning process in which the results of one test

suggest additional tests• We will explore Exploratory Testing with an example.


James Bach’s Exploration Mindset“I run a test.

I MAKE AN OBSERVATION...

- I experience surprise associated with a pattern within an observation.

That triggers REFLECTION about PLAUSIBILITY...

- The pattern seems implausible relative to my current model of thephenomenon.

That triggers REFLECTION about RISK...

- I can bring to mind a risk associated with that pattern.

That triggers REFLECTION on MAGNITUDE OF RISK...

- The risk seems important.

That triggers TEST REDESIGN... ”


Exploratory Testing DynamicsCreated by James and Jonathan Bach v1.2 Copyright © 2005, Satisfice, Inc

Chartering. Making your own decisions about what you will work on and how you will work.

Manipulating. Making and managing contact with the object of your study; configuring and interacting with it. Designing experiments and establishing lab procedures.

Observing. Gathering empirical data about the object of your study; collecting different kinds of data, or data about different aspects of the object. Designing experiments and establishing lab procedures.

Modeling. Composing, describing, and working with mental models of the things you are exploring. Identifying relevant dimensions, variables, and dynamics. A good mental model may manifest itselfas having a “feel” for the product; intuitively grasping how it works.

Conjecturing. Considering possibilities and probabilities. Considering multiple, incompatible explanations that account for the same facts.


Exploratory Testing Dynamics (continued)Created by James and Jonathan Bach v1.2 Copyright © 2005,

Satisfice, Inc

Questioning. Identifying missing information, conceiving of questions, and asking questions in a way that elicits the information that you seek.

Recording. Preserving information about your process, progress, and findings. Reporting. Making a credible, professional report of your work to your clients in

oral and written form.Pairing. Working and thinking with another person on the same problem.Resourcing. Obtaining tools and information to support your effort. Exploring

sources of such tools and information. Getting people to help you.Refocusing. Managing the scope and depth of your attention. Looking at

different things, looking for different things, in different ways. Branching/Backtracking. Allowing yourself to be productively distracted from

one course of action in order to explore an unanticipated new idea.Generating/Elaborating. Working quickly in a manner good enough for the

circumstances. Revisiting the solution later to extend, refine, refactor or correct it.

Alternating. Switching among different activities or perspectives to create or relieve productive tension and make faster progress.


Exploring the Commission ProblemLock, Stock, and Barrel

In the Commission Problem, a rifle salesman in the old American West sells rifles, either as a whole, or in the three parts, the lock, the stock, and the barrel. Prices are:

Locks: $45Stocks: $30Barrels: $25Complete Rifle: $100

The salesman’s commission is computed on a sliding scale:10% on the first $1000 of total sales15% on sales from $1001 to $180020% on sales over $1800


Recognition of a FaultIn the first few months, the salesperson never

exceeds $1000 in sales, and the commission is correct. When sales finally reach the 15% level, however, the salesperson’s commission is less than expected. And one month, when sales reached $1900, the commission was slightly more than expected.

If the salesman had a spreadsheet, he might have tried out some possible sales…


The First Exploration

($20)fail$260 $240 $1,9001919197

$15 fail$205 $220 $1,8001818186

$15 fail$190 $205 $1,7001717175

$15 fail$100 $115 $1,1001111114

$0 pass$100 $100 $1,0001010103

$0 pass$80 $80 $8008882

$0 pass$10 $10 $1001111

Expected less

ComputedPass?Computed

commissionExpected

commissionsalesbarrelsstockslockscase

#


The Exploration Begins

The rifle salesman knows the computation involves four business rules:

(1) sales = $45*locks + $30*stocks + $25*barrels

(2) commission = 0.10*sales (for $0 <= sales <= $1000)

(3) commission = $100 + 0.15*(sales - $1000)(for $1000 < sales <= $1800)

(4) commission = $220 + 0.20*(sales - $1800)(for sales > $1800)

The sales calculation is correct, but could there be offsetting errors in the coefficients?


The Second Exploration

$0.00pass$25.00$25.00$250.00100010

$0.00pass$30.00$30.00$300.0001009

$0.00pass$45.00$45.00$450.0000108

Expected less

Computed

PassFail?

Computed commission

Expected commission

salesbarrelsstockslockscase #

Try calculating with one variable at a time.

Looks OK. No problem with the coefficients, so Equation (1) is being computed correctly. The salesman already knows that equation (2) is also being computed correctly. Time to explore equation (3).


The Third Exploration

$15.00 fail$85.75 $100.75 $1,005.022113

$0.00 pass$100.00 $100.00 $1,000.112112

$0.00 pass$99.50 $99.50 $995.202111

Expected less

Computed

Pass/Fail?

Computed commission

Expected commission

salesbarrelsstockslockscase #

Check Equation (3). Try to devise sales near the $1000 level.

Aha! (Ein Licht geht auf!) the only thing wrong with equation (3) must be the amount subtracted from sales. Solving equation (3) with the computed and expected commission: (3 expected) $100.75 = $100 + 0.15($1005 - $1000)(3 computed) $85.25 = $100 + 0.15($1005 – x),

x = $1100


Not Finished Yet

There is a remaining fault. Look at test case 7, in which the salesman receives too much commission. You can try exploratory testing with the exploreCommission.xls spreadsheet.


Observations and Conclusions

• Exploratory testing requires discipline• Not an ad hoc activity• Documentation, especially of intermediate results, is

valued• Allows a tester to focus on “deep” faults• Can be extremely effective• Can be extremely efficient• Very dependent on the tester


Test-Driven Development

• The essence of agility• Also known as Test-First Development• Requires automation• A bottom-up process• Presumes refactoring• Excellent for fault isolation• Test “granularity” is an issue• Big Question: does TDD result in a good

design?


Test-Driven Development (TDD) Life Cycle

Create a test

Run the fulltest set

Write code so test passes

RefactorAs needed

Start


TDD Example: a Boolean Function to Determine Leap Years

• Definition: A year is a leap year if it is a multiple of 4, but century years are leap years only if they are multiples of 400.

• Test-Driven Development would break this into small, individual user stories (also called tasks).

• “Coded” here in a pseudo-code (a lingua franca) that resembles Visual Basic.

• Try it yourself (with no looking ahead) in your favorite programming language.


My isLeap Code


User Story 1: A year divisible by 4 is a leap year

Test Case 1Input: 2004 Expected Output: True

(existing) Pseudo-Code in normal fontFunction isLeap(year) As BooleanEnd isLeap

Running Test Case 1 on this code fails.Add just enough code to make the test pass.


User Story 1: A year divisible by 4 is a leap year

Test Case 1Input: 2004 Expected Output: True

(updated) Pseudo-Code in bold face fontFunction isLeap(year) As Boolean

dim year AS IntegerisLeap = False'MOD is the modulo arithmetic built-in operator in most languagesIf (( year MOD 4) = 0) Then

IsLeap = TrueEndIf

End isLeap

Test Case 1 passes. Now do User Story 2.


User Story 2: A year not divisible by 4 is a common year

Test Case 1 Input: 2004 Expected Output: True

Test Case 2 Input: 2007Expected Output: False

(existing) Pseudo-Code in normal fontFunction isLeap(year) As Boolean

dim year AS IntegerisLeap = FalseIf (( year MOD 4) = 0) Then

IsLeap = TrueEndIf

End isLeap

Test Case 1 passes. Test Case 2 fails. Now add just enough code so that Test Case 2 passes.


User Story 2: A year not divisible by 4 is a common year

Test Case 1 Input: 2004 Expected Output: True

Test Case 2 Input: 2007Expected Output: False

(updated) Pseudo-Code In bold face fontFunction isLeap(year) As Boolean


IsLeap = TrueElse

IsLeap = TrueEndIf

End isLeap

Test Cases 1 and 2 pass. Now do User Story 3


User Story 3: A century year not divisible by 400 is a common year

Test Case 1 Input: 2004, Expected Output: TrueTest Case 2 Input: 2007, Expected Output: FalseTest Case 3 Input: 1900, Expected Output: False

(existing) Pseudo-Code In normal fontFunction isLeap(year) As Boolean


IsLeap = TrueElse

IsLeap = TrueEndIf

End isLeap

Test Cases 1 and 2 pass. Test Case 3 fails. Now add just enough code so that Test Case 3 passes.


User Story 3: A century year not divisible by 400 is a common year

Test Case 1 Input: 2004, Expected Output: TrueTest Case 2 Input: 2007, Expected Output: FalseTest Case 3 Input: 1900, Expected Output: False


dim year AS IntegerisLeap = FalseIf ((( year MOD 4) = 0) AND NOT((year MOD 100) = 0))) Then

IsLeap = TrueElse

IsLeap = TrueEndIf

End isLeap

Test Cases 1, 2 and 3 pass. Now do User Story 4


User Story 4: A century year divisible by 400 is a leap year

Test Case 1 Input: 2004, Expected Output: TrueTest Case 2 Input: 2007, Expected Output: FalseTest Case 3 Input: 1900, Expected Output: FalseTest Case 4 Input: 2000, Expected Output: True

(existing) Pseudo-Code In normal fontFunction isLeap(year) As Boolean

dim year AS IntegerisLeap = FalseIf ((( year MOD 4) = 0) AND NOT((year MOD 100) = 0))) Then

IsLeap = TrueElse

IsLeap = TrueEndIf

End isLeap

Test Cases 1, 2 and 3 pass. Test case 4 fails. Now add just enough code so that test case 4 passes.


User Story 4: A century year divisible by 400 is a leap year

Test Case 1 Input: 2004, Expected Output: TrueTest Case 2 Input: 2007, Expected Output: FalseTest Case 3 Input: 1900, Expected Output: FalseTest Case 4 Input: 2000, Expected Output: True


dim year AS IntegerisLeap = FalseIf ((( year MOD 4) = 0) AND NOT((year MOD 100) = 0))) OR

((year MOD 400 = 0)) ThenIsLeap = True

ElseIsLeap = True

EndIfEnd isLeap

Test Cases 1, 2, 3 and 4 pass. Done with function isLeap.


Advantages of Test Driven Development

• In this example, the steps are deliberately small.• Customer and developer can (should!) jointly

determine granularity of user stories.• Fault isolation is greatly simplified (in fact, trivial). If

a test case fails, the fault must be in the most recently added code.

• Once a new test case passes, a working (subset) of the desired software can always be delivered.

• Something always works!


Disadvantages of Test Driven Development

• Useful granularity is an issue.• There is no guarantee that user stories “arrive” in a

sensible order.• There is no guarantee that user stories are the

“same size” (or require similar effort)• Bottom-up coding often results in poorly structured

code, making refactoring necessary.• Tool support (zB jUnit) is essential.


Comparison of Model Driven Development (MDD) and Test Driven Development (TDD)

• First American’s view of Eagles and Mice– Eagles have the “big picture”– Mice focus on the details– (both views are important!)

• MDD is a rigorous, top-down approach.

• TDD is a bottom-up approach.


TDD isLeap in Visual Basic (refactored)

1

2

3

4

5

6 7

8

9

Public Function isLeap(year) As BooleanDim year As IntegerDim c1, c2, c3 As Boolean

1. c1 = (year Mod 4 = 0) 2. c2 = (year Mod 100 = 0) 3. c3 = (year Mod 400 = 0) 4, isLeap = False5. If ( (c1 AND NOT(c2)) OR (c3)) Then6. IsLeap = True7. Else8. IsLeap = False9. EndIfEnd Function


Decision Table Model of isLeap

2007200419002000test case: year =

XXA2. year is a leap year

xXA1. year is a common year

XXXX(logically impossible)

Actions

FTFTFTFTC3. year is a multiple of 400

FFTTFFTTC2. year is a century year

FFFFTTTTC1. year is a multiple of 4

r8r7r6r5r4r3r2r1Conditions


MDD isLeap in Visual Basic1

11

12

13

14

15

16

17

2

3

4

5

6

7

8 9

10

Public Function isLeap(year) As BooleanDim year As IntegerDim c1, c2, c3 As Boolean

1. c1 = (year Mod 4 = 0) 2. c2 = (year Mod 100 = 0) 3. c3 = (year Mod 400 = 0) 4, isLeap = False5. If c1 Then6. If c2 Then7. If c3 Then 8. isLeap = True ‘rule r19. Else10. isLeap = False ‘rule r211. End If12. Else13. isLeap = True ‘rule r414. End If15. Else16. isLeap = False ‘rule r817. End IfEnd Function


Observations

• The TDD version is less complex (really?)– Cyclomatic complexity of the DD version is 2– Cyclomatic complexity of the MDD version is 4

• The TDD version gradually built up to a compound condition (that might be hard to understand, and to modify).

• The decision table model assures– Completeness– Consistency– Non-redundancy

• Both versions require 4 test cases


Open Questions

• Does Agile (bottom up) Development actually result in better designs?

• Can any form of Agile Testing reveal “deep faults”?– zB: faults revealed only by data flow (define/use) testing– computational faults– time-dependent faults

• What about maintenance? Agile code has– well-named variables and components– (hopefully) been refactored carefully– no comments– test cases are the specification


Conclusions• The Agile community is VERY passionate about their

methods.• Agile Development clearly works well for SOME

applications– time critical– uncertain customer (I’ll know what I want when I see it!)

• Some agile methods apply to non-agile projects– Context-Driven Testing – Good Enough Testing – Exploratory Testing

• Test Driven Development is excellent (ausgezeichnet!) for fault isolation.

• Try it yourself with tddTriangle.xls


Also geht alles zu Ende allhier:Feder, Tinte, Tobak auch wir.Zum letzten Mal wird eingetunkt,dann kommt der große schwarze

W. Busch

Documents

Agile Testing Workshop - itq.chitq.ch/pdf/sqm/07/ics_jorgensen_agile_testing.pdf · Agile Testing Workshop SQC 2007, Düsseldorf Paul C. Jorgensen, Ph.D. The Agile Manifesto “We