ATDD - Acceptance Test Driven Development

Licensed Under Creative Commons by Naresh Jain

Acceptance Test Driven Development

Naresh [email protected]

http://blogs.agilefaqs.com

1

http://creativecommons.org/licenses/by-nc-sa/3.0/


http://agilefaqs.com/nareshjain.html


mailto:[email protected]

mailto:[email protected]




Warmup Scenarios

2






Warmup ScenariosPick one scenario and in relation to your scenario, what are the specific observable results that will tell you that the activity has been successfully completed?

2







Going out for Movie (THX sound and Digital projection)

2








Going out for meal (one veg.)

2









Going shopping ($50)

2









Going shopping ($50)[10 Minutes]

2









Going shopping ($50)[10 Minutes]

Present back to the group your findings. [3 minutes per group]

2





Story is a smallest piece of functionality that add business value

What is a Story?

Ron Jeffries’ 3 Cs - Card, Conversation and Confirmation

As a .. <user who requires this feature>

I want .. <do something>

So that... <user goal/business justification>

Story Title - Actor Action Context

3

Title: Keen Reader subscribes to a blog

As a keen reader of your blog

I want to subscribe to your blog

So that I can stay up-to-date with the new posts

Story Example

4

Title: Social Networking Enthusiast uploads profile picture

As a Social Networking Enthusiast

I want to upload my profile picture

So my friends can see how I look and recognize me

Another Story Example

5

What makes a good Story?

6

Stories should follow the INVEST principle:


6


Independent


6


Independent

Negotiable


6


Independent

Negotiable

Valuable


6


Independent

Negotiable

Valuable

Estimate-able


6


Independent

Negotiable

Valuable

Estimate-able

Small


6


Independent

Negotiable

Valuable

Estimate-able

Small

Testable


6

Stories are fundamental unit of activity

7

Business Goals


7

Business Goals

Inception


7

As a ____, I want to be able to ____ so

that ____

Product Backlog

Might have an initial estimate (perhaps for both analysis and development), and an expression of technical and

business confidence that this is real and achievable

Business Goals

Inception


7


that ____

Product Backlog



Release planning

Business Goals

Inception


7


that ____

Product Backlog




that ____

I will know this is done when _______

Release Backlog

More detailed estimate, and a specific acceptance test – low confidence stories might be

“spiked” or prototyped

Release planning

Business Goals

Inception


7


that ____

Product Backlog




that ____


Release Backlog



Release planning Sprint planning

Business Goals

Inception


7


that ____

Product Backlog




that ____


Release Backlog




that ____


To do this I must:1) _____2) _____

Sprint Backlog

Possible automation of the acceptance

test

Development team breaks out the detail of work needed to

pass test

Release planning Sprint planning

Business Goals

Inception


7

Is a set of conditions that the Story must meet for it to be accepted as complete

Is typically provided by the customer or product owner.

Is not a replacement for conversation.

Is the results of the conversation

Acceptance Criteria are NOT tests

Acceptance Criteria

8

Acceptance Criteria should contain:

ACTOR

VERB – DESCRIBING A BEHAVIOR

OBSERVABLE RESULT

To accommodate pre-conditions Acceptance Criteria can be expressed as

Given [Precondition]

When [Actor + Action]

Then [Observable Result]

Writing Acceptance Criteria

9

ExampleSocial Networking Enthusiast uploads profile picture

10

Example

Given the user has a valid facebook account and a digital picture on her computer,

When she uploads a picture in facebook,

Then her the picture should be visible to all her friends in her network.

Social Networking Enthusiast uploads profile picture

10

Example




Given an user is trying to find a friend on facebook,

When the user searches for a person using their name,

Then their profile picture should be displayed along with other details.


10

Example




Given an user is trying to find a friend on facebook,

When the user searches for a person using their name,

Then their profile picture should be displayed along with other details.

As owner of facebook,

I want users to upload authentic, personal profile picture,

So facebook's reputation remains intact and facebook stays out of legal hassles.


10


Acceptance TestsAcceptance Criteria & Tests: Definition

11







Acceptance Criteria

11







Acceptance Criteria

+

11







Acceptance Criteria

Examples (data + scenarios)+

11







Acceptance Criteria

Examples (data + scenarios)

Acceptance Tests

+

11





TasksTeam members further break down each story into tasks that need

to be completed to meet the acceptance criteria for the story.

12

Tasks

To accomplish this story:

we start off with a simple upload and image display

Team members further break down each story into tasks that needto be completed to meet the acceptance criteria for the story.

12

Tasks



restrict user to only upload certain image types (gif, jpg and png)


12

Tasks




figure out where to store the image. (performant and fault-tolarent)


12

Tasks





scale down (size, resolution, etc.) of the image


12

Tasks





scale down (size, resolution, etc.) of the image

and so on...


12


DemoRoman Numerals to Decimal Conversion Example

13






DemoReal World Domain Forwarding Server

14






Thinking in Tables

15






Only Tables ExecuteThinking in Tables

Ignored

Executed

16






Foundational Table Structure

Name of Fixture

Interaction with Application

Table structure depends on type of Fixture

Thinking in Tables

17






3 Foundation Fixtures

Column Fixture

Row Fixture

Action Fixture

Thinking in Tables

18






Column FixtureThinking in Tables

19






Row FixtureAnalogous to comparing against rows in a database table

Thinking in Tables

20






Action Fixture

Think GUI window Counter WindowCounter:

Counter: 6 Count

public class CountFixture extends Fixture { private int counter = 0; public void count() { counter++; } public int counter() { return counter; }}

Thinking in Tables

21






FitLibrary

Extension to FIT

Written by Rick Mugridge

Adds some handy Fixtures

22






FitLibrary Fixtures

ArrayFixture for ordered lists

SetFixture for unordered lists

SetUpFixture

Supports

Graphics

Tree structures

Nested Tables

FitLibrary

23






DoFixture

• Broken tables

• Highly readable

• Flexibility

FitLibrary

24






FIT

Framework for Integrated Tests

Created by Ward Cunningham

Open Source

The most accepted solution for agile acceptance testing

Tools

25






FitNesse

Environment build around FIT

Makes everything easier

Created by Object Mentor, Inc.

Open Source

Tools

26






FIT FitNesse

- Tests written in HTML- Tests are executed on the command line- Tables are executed- Non-table markup is ignored- Tables map to Fixtures- Fixtures are code that is aware of the system- Supplies foundational Fixtures- Implementations ported to many languages

- Stand alone web server- Is a wiki- Tests written in wiki text- Tests are executed from within the wiki- Translates tests into HTML- Uses FIT to execute tests- Supports test suites- Supports variables in tests- Supports test refactoring- Written in Java- Supports FIT implementations in any language

27






Acceptance Criteria and Tests:

A Critical Piece of Agile

28






Analysis

Design

Implementation

DFDERD

DDST

1 May 1 Nov1 Jul 1 Sep

Traditional ApproachAcceptance Criteria & Tests: A Critical Piece of Agile

29






Key Questions

Are we building the right product?

Are we building the product right?

Business Facing

Technology/Implementation Facing

30






Brian Marick’s Test Categorization

Business Facing


Supp

orts

Pro

gram

min

gC

ritique product

31






It Helps to Think of Tests this way...

Business Facing


Dri

ves

Dev

elop

men

t Critique product

32







Business Facing


Dri

ves

Dev

elop

men

t Critique productUnit Testing

32







Business Facing


Dri

ves

Dev

elop

men


Acceptance Testing Low-fi prototypes

32







Business Facing


Dri

ves

Dev

elop

men



Exploratory TestingUI and Usability Testing

32







Business Facing


Dri

ves

Dev

elop

men



Exploratory TestingUI and Usability Testing

Performance TestingSystem Tests

32






Avatars of TDD

Business Facing


Dri

ves

Dev

elop

men

t Critique product

Outside In

Inside Out

33






TDD Rhythm - Test, Code, Refactor

Test Driven Development

34







Add a Test


34







Add a Test

Run the Test


34







Add a Test

Run the TestPass


34







Add a Test

Run the Test

Fail

Pass


34







Add a Test

Run the Test

Make a little change

Fail

Pass


34







Add a Test

Run the Test


Run the Test

Fail

Pass


34







Add a Test

Run the Test


Run the Test

Fail

Fail

Pass


34







Add a Test

Run the Test


Run the Test

Fail

Pass

Fail

Pass


34







Add a Test

Run the Test


Run the Test

Refactor

Fail

Pass

Fail

Pass


34







Add a Test

Run the Test


Run the Test

Refactor

Fail

Pass

Fail

Pass


34







Add a Test

Run the Test


Run the Test

Refactor

Fail

Pass

Fail

Pass


34







Add a Test

Run the Test


Run the Test

Refactor

Fail

Pass

Fail

Pass


34







35







Story

35







Story

Acceptance Criteria

35






Acceptance Test Driven DevelopmentIteration

Story

Acceptance Criteria

35







Automated Acceptance

Tests

Story

Acceptance Criteria

35








Tests

Story

Acceptance Criteria

Automated Unit Test

35








Tests

AutomatedAcceptance

Tests

Story

Acceptance Criteria

Automated Unit Test

35








Tests

AutomatedAcceptance

Tests

Story

Acceptance Criteria

ExploratoryTesting

Automated Unit Test

35








Tests

AutomatedAcceptance

Tests

Story

Acceptance Criteria

Acceptance Criteria

ExploratoryTesting

Automated Unit Test

35








Tests

AutomatedAcceptance

Tests

Story

Acceptance Criteria

Acceptance Criteria

ExploratoryTesting

Automated UI Tests

Automated Unit Test

35








Tests

AutomatedAcceptance

Tests

Story

Acceptance Criteria

Acceptance Criteria

ExploratoryTesting

Automated UI Tests

TESTS

PERFORMENCE

Automated Unit Test

35






Mike Cohn’s Testing Pyramid

36







GUITests

36







GUITests

Small in NumberTools: Selenium, Sahi, Watir, Abbot, Frankenstein

36







AcceptanceTests

GUITests


36







AcceptanceTests

GUITests


At least one per storyTools: Fit, FitNesse, RSpec, JBehave

36







Unit Tests

AcceptanceTests

GUITests



36







Unit Tests

AcceptanceTests

GUITests



At least one per class or moduleTools: xUnit, TestNG

36






Every story must have at least one Acceptance Test

A story is not DONE until it passes it’s Acceptance Tests

Criteria for DONEAcceptance Tests: A Critical Piece of Agile

37






Manual Acceptance TestsAcceptance Tests: A Critical Piece of Agile

38






Manual Acceptance Tests


Acceptance Tests: A Critical Piece of Agile

38









38






Why Acceptance Tests?

Criteria for Completion

Great Collaboration tool

Source of Feedback

Real data to measure progress

39






Data From Acceptance Tests

0

18

36

54

72

90

1 2 3 4 5 6 7 8 9 10

Total ATs Failing ATs Passing ATs

40






Acceptance Tests Are Automated

41






The Button

How often would you press it?

When would you press it?

Who would press it?

Testers, Developers, Managers, Customers, Spectators, etc.


42





Licensed Under Creative Commons by Naresh Jain43






Criteria for DONE

43






Criteria for DONE

+

43






Criteria for DONE

Automated+

43






Criteria for DONE

Automated

Executable Specification

+

43






Executable Specification

A new paradigm for testing

Puts quality first

Removes ambiguity from requirements


44






Who Writes Acceptance Tests?

45






The Customer

The Customer Role

Stake holder

Business Analyst

Quality Assurance

Product Owner

Developer


46






Tests Get Technical

The “Customer” may need technical help to write tests

Developers and QAs are technical

Pair test authoring


47






Business Rules Get Fuzzy

Sometimes developers need help understanding tests

Customers know business rules

Pair test implementation


48






Exercise #1

49






The Login Test

Write a test plan, in plain text, for the business rules of logging in.

Web application

User credentials are stored in relational database

Successful login redirects to “Welcome” page

Exercise #1

50






Writing Good Acceptance Tests

51






Login Test PossibilitiesWriting Good Acceptance Tests

1. Direct browser to URL for login page

52








52








1. Enter the username ‘wallace’

52









52









Build a Testable Environment First

52






Login Test Possibilities

1. Add some users to the system


53









53








3. Enter a value into the username field


53










53









Be Specific


53






Tests are Examples

Use concrete examples

Specify concrete behavior

No ambiguity allowed


54







1. Insert into User table values (’wallace’, ‘ilikecheeze’)


55









55








2. Open a browser to the URL http://localhost/myapp


55





http://localhost/myapp









55













Avoid Implementation Details


55










Good Acceptable Criteria and Tests

56







S - SPECIFIC - Explicitly defined and definite

56








M - MEASURABLE - Possible to observe and quantify

56









A - ACHIEVABLE - Capable of existing or taking place

56










R - RELEVANT - Having a connection with the story

56










R - RELEVANT - Having a connection with the story

T - TIME-BOUND – When will the outcome be observed

56






Avoid Implementation Details

Business Tier

View

Data Store

Acceptance Tests

DatabaseDatabase

UI


Model and Presenter

57






Login Test: Possible Solution

Add user to system: (’wallace’, ‘ilikecheeze’)

Process login with username ‘wallace’ and password ‘blah’

Check login failed

Process login with username ‘wallace’ and password ‘ilikecheeze’

Check login succeeded


58






Tools

59






Commercial Tools

WinRunner

Silk

RFT

Tools

Are not suitable for Acceptance Testing in an Agile environment

TestPartner

QTP

Squish

EggPlant

TestComplete

WindowTester

60






Open Source Options

FIT

FitNesse

Selenium

Among the few tools that support Test Driven Development

Sahi

Watir

Abbot

Frankenstein

Cucumber

RSpec/JBehave

Tools

61






Wiki

62






What is it?

A collaborative web site

Editable by any

Created by Ward Cunningham

Every project should have one

http://c2.com/wiki

http://en.wikipedia.com

Wiki

63





http://c2.com/wiki

http://c2.com/wiki




Creating Tests

Use Wiki syntax to create a page with test tables Label the page as a Test Page

Use a page name of the form Test… Turn on the Test property

Make sure your Fixtures are in the classpath Use !path widget

Mechanics !path values are concatenated Java command to start FitServer is executed Testable HTML is passed to FitServer FitServer runs the tests Results are passed back to FitNesse

Wiki

64






Creating Suites

Set the Suite property Create a page with the Suite property Created test pages inside this page When the suite is executed, all child test pages will

be included in the suite execution Use the !see widget

!see <name of test page> All “included” tests pages will be included in the

suite execution

There are 2 ways to make Suites

Run a Suite by clicking the Suite button

Wiki

65






Hands-on Session

Conference Proposal Submission Portal

Some sample Stories

Should be able to submit new proposalShould be able to list all submitted proposalSubmitting proposal with same title should display appropriate error messageShould be able to delete submitted proposal based on the title Should be able to delete submitted proposal based on the titleShould be able to search proposals by titleShould be able to search proposals by IDShould be able to find all proposal by an author's name

66






Break

67






Patterns

68






Organizing Tests

Allowing customers to add new tests without breaking the build

Patterns

69






Version Control

Keeping the acceptance test in version control with the code.

Patterns

70






Cross-Functional Pairing

Using FitNesse based acceptance tests for collaboration between cross-functional team members.

Patterns

71






ATDD


Patterns

72






CSTT

Cleanup, Setup, Test, Teardown

Patterns

73






Independent Tests

Tests shouldn’t depend on each other.

Tests leave the system in the same state it started in.

Patterns

74






Dynamic Stubbing

Avoiding complications of external systems.

Patterns

75






Non-Production Setup/Teardown

Using non-production light weigh code for setup and teardown.

Helps test only what you want to test.

Patterns

76






Suite Levels

Creating different levels of suites depending on the depth/level of feedback desired.

Smoke, Current Iteration/Sprint, Regression

Patterns

77






DRY

Using !include to avoid repeating yourself.

Patterns

78






Make it Real

Write ATs as close as possible to the real environment.

Patterns

79






Fixture Evolution

Allow Fixture implementation to evolve over time.

Treat fixtures as first class citizens.

Patterns

80






At Least One Test/Story

Every story should have at least one acceptance test

Avoid long/multipurpose tests.

Patterns

81






Anti-Patterns

82






Developer ATs

Developers writing acceptance tests by themselves, for themselves.

Anti-Patterns

83






Unit Testing

Don’t write ATs at the unit testing level

Unit tests are implementation specific

ATs are NOT implementation specific

Anti-Patterns

84






QA Testing Tool

Hard to write tests up front.

Perhaps only on large projects.

Anti-Patterns

85






Silver Bullet

Trying to use FitNesse for all types of Acceptance Tests

UI testing

XML testing

Anti-Patterns

86






Test After

Writing tests after the code is already written.

Anti-Patterns

87






Hidden Test Data

Hiding test data in the fixtures.

Anti-Patterns

88






Implementation Dependent ATs

Making test pages (tables) dependent on implementation details and data structures.

Anti-Patterns

89






Logging in Your Fixtures

Putting log statements or print statements in the fixture code.

Fixtures are probably too complicated.

Anti-Patterns

90






Reference

Portions of this presentation is adopted from Micah Martin’s Introduction to Automated Acceptance Tests Presentation

Kent Beck, Test Driven Development By Example.

"Agile Testing Directions" - Brian Marick

http://www.opensourcetesting.org/

91





http://www.testing.com/cgi-bin/blog/2003/08/21%23agile-testing-project-1

http://www.testing.com/cgi-bin/blog/2003/08/21%23agile-testing-project-1

http://www.opensourcetesting.org

http://www.opensourcetesting.org


The End

[email protected]://blogs.agilefaqs.com

Questions?

92





http://www.objectmentor.com


