Workshop on Agile Test Strategies and Experiences...‘The nature of the tester's role changes in iterative projects. We are no longer the high-profile victims, we are no longer the

Workshop on AgileTest Strategies

and Experiences

Fran O'Hara, Insight Test Services,

IrelandW8

Copyright Insight Test Services 2007 2

Europe’s leading independent specialist software

QA, Test & Compliance full services provider

www.insight-test.com

Workshop – Agile Test

Strategies and ExperiencesVersion 1.0

Fran O’Hara

[email protected]


We help ensure compliance

(e.g.Business needs, Best Practice in Software

Quality and Test, CMMI/TMMi, SOX, FDA, etc.)

In a practical way we help

•Over 75 Organisations and growing using

•90+ software QA & Test experts with Businessand Regulatory expertise

•Test Control™ (Best Practice methodology)

•Full Life Cycle capability

Business and technology expertise•Financial Services

•Public Sector

•Information & Comms. Technology

•Life Sciences

•Transport/utilities

ServicesHelping organisations

achieve their software

quality, compliance and

delivery goals.

• Consulting– Process audits/assessments

– Process improvement

– Regulatory Compliance (SOX,

Mifid, FDA, etc.)

– Test (automation) strategy

– TestControl™ our risk-based

test methodology

– Vendor quality management


• Training & Competence Development - extensive

range of topics including :

– CMMI, TMM(i), TPI

– ISEB / ISTQB

– Risk-based testing, UAT, etc.

• Managed Test Services

– On-site, near shore, far shore

– System (non-)funct., UAT…

– Test automation services

• Specialist Resourcing

– Specialist skills, short/long

term & Supplementary

resources


Workshop Objectives

• Identify typical challenges in agile

development

• Present/discuss test strategies in

agile environments

• Discuss approaches and

experiences in resolving

challenges and issues identified


Your Challenges


Some Typical Challenges 1/2

• Testing issues around iterative/incremental development versus traditional development

• Hybrid approaches

• Role of the tester

• Levels/phases of test – what does ‘acceptance testing’ mean?

• Effectiveness of automated tests


Some typical challenges 2/2

• System Testing– When is it done?

– Who does it?

– What’s delivered?

• Project Management– Estimate, resources, costs, time, etc

• Change Management

• Documentation e.g. testing without detailed requirements

• Scale

• Multi-site teams and tester integration


Agenda Items

• Agile test overview– Different Test Phases, agile terminology

– Where the different types of testing fit into an agile development lifecycle

• Role of the tester

• Test Driven Development (TDD), Automated Unit Testing

• Acceptance testing, testing and the customer/user

• Exploratory testing


Agile Testing - overview


Agile Values

“We are uncovering better ways of developing

software by doing it and helping others do it.

Through this work we have come to value:

– Individuals and interactions over processes and

tools

– Working software over comprehensive

documentation

– Customer collaboration over contract negotiation

– Responding to change over following a plan

That is, while there is value in the items on the right,

we value the items on the left more.”


Agile Methodologies

• eXtreme Programming (XP)

• SCRUM

• DSDM

• Crystal

• Lean Development (Manufacturing)

• Adaptive Software Development

• Feature Driven Development

• Context Driven Development


Remember…there are no best

practices – context is key


Agile View of Traditional

Development

Requirements Analysis Design DevelopmentSystem

Test

UAT

Support

Post

Project

Delivery of System

- Stage at which defects cause most problems

Time

Co

st


Traditional Testing

• Static– Requirements & Design Reviews

– Code Reviews

– Static Analysis of Code

• Dynamic– E.g. Unit, Unit Integration, System,

System Integration, UAT.

– Functional

– Non-functionalCopyright Insight Test Services 2007 16

V-Model


Requirements

Functional Spec.

Hi level design

Acceptance test

Lo level design

Code

System test

Integration test

Unit test

Reviews

Static Analysis

Static Testing Dynamic Testing

Early test

design

Agile Development

A D Dev AT

Time

Co

st

A D Dev AT A D Dev AT

Deliver something of value to the customer


Note the tradeoff (with iteration):

Features versus time, not quality (and testing) versus time.

Focus Of Agile Testing• High Value Features First

• Test Driven Development

• Automation of Unit Testing

• Continuous Integration

• Pair Programming– Independent Reviews

– Independent Testing

• Automated Acceptance Testing

• Constant regression

• Exploratory Testing

• Multiple Builds – Delivering Value


Common Issues in Agile

Testing

• Attempting to substitute unit tests for

acceptance tests

• Not automating the acceptance tests

• Thinking the automated tests are

sufficient

• Forgetting about non-functional tests.

– Quality characteristics such as performance,

reliability, usability scalability, memory usage,

etc.


See www.testdriven.com

Role of the Tester

• Iteration and Variable scope…– Have a huge impact on a test department!

• If development is iterative…– When do you test?

– What do you test?

– How do you test?

• If scope is variable…– What do you test?

– When do you write the tests?


‘The nature of the tester's role changes in iterative projects. We are no longer the high-profile victims, we are no longer the lonely advocates of quality, we are merely (!) competent service providers, collaborating with a group that wants to achieve high quality. ‘

Cem Kaner

Dedicated testers bring two benefits:

• Focus on customer usage over technical implementation

• Focus on uncovering flaws over confirming completeness

(Bret Pettichord)


Role of tester

• Fully integrated into team

• Involved continuously from start…e.g.– Facilitate communication between the technical &

business stakeholders

– Support early validation of requirements

– Help the business stakeholders define acceptance criteria

– Create automated acceptance tests

– Expand scope of ‘acceptance’ tests

– Advise the team about overall risks and trends

– Perform manual/exploratory tests on early-stage code

• May need to develop scripting skills


Role of tester

Test Driven Development

• Never write a single line of code unless you have a failing automated test

• Eliminate duplication

Kent Beck

• Write the test

• Write the code

• RefactorCopyright Insight Test Services 2007 24

Test Driven Development• Can be applied at all levels of test e.g.

Unit and Acceptance

• Preventative/early testing – not new… but many benefits

• Testing takes on a specification role…

…not a verification role?– A feature is not specified…

• Until it’s acceptance test is written.

– A feature is not done…• Until all it’s acceptance tests pass.

– Acceptance and Unit tests become key requirements/feature and design artefacts


Unit level automation

• Automation Of Unit Test– More likely to be done

– As development is done

– Structural coverage measurement

– Daily unit test regression providing stability with high level of change/iteration

• Need to engineer unit test code with same discipline as application code

• ‘Coverage complacency’ & ‘Happy path testing’


Acceptance Testing

• Automated Acceptance Testing– Design and Code for automation

– Automation Frameworks e.g. • Exactor, Brian Swan & Sean Hanly

http://www.exoftware.com/xp_tools.htm

• FIT, Ward Cunningham, http://fit.c2.com/

• AXE, Odin Technology, http://www.odin.co.uk/

– Issues• Can be too low level or not business focused

enough

• Thin UI layer issue


Exactor Example


# Test zero initially

StartCalculator

CheckResultEquals 0

# Test addition

Add 4 5 1

CheckResultEquals 10

# Test subtraction

Subtract6

CheckResultEquals 4

Subtract 1 2

CheckResultEquals 1

# more tests...

• Based on Junit

and JWebUnit

•Freely Available

•Tester defines

Tests using

Keywords

•Automation Eng

writes code

behind Keywords

FIT Example


• Executes Scripts

scripted as tables

in a html

document

•Will interpret html

differently

depending on

Fixture used

•See also FitNesse

and FitLibrary


From: ‘Home Brew test automation’ – Bret

Pettichord

Acceptance Testing – is it

enough?

• May not be…context/risk/strategy issue…– May not be fully automated – partial regression

strategy needed

– Expand to fuller ‘system’ tests

• Functional testing

• Non-functional testing – performance, usability, etc.

– May still need end-to-end business scenario focused User Acceptance test

– System integration testing issues

– Etc.

• Strategy and scheduling issue– Adaptive, risk-driven


Iteration testing

Copyright Insight Test Services 2007 32From Bret Petticord – challenges of agile testing

Exploratory testing

• Simultaneous test design, execution

and learning

• Exploratory Testing

– Experienced-based technique

– Usually mixed with and complements

documented test cases

– Creative Process


Exploratory testing

“… agile programs are more subject to unintended consequences of choices simply because choices happen so much faster. This is where exploratory testing saves the day. Because the program always runs, it is always ready to be explored."

Ward Cunningham on Why should agile teams do exploratory testing?:


Exploratory testing

Can Use test automation to support early exploratory testing.

Traditional test wisdom says we can’t start

testing a feature until it’s accessible from

an external interface (like a GUI). But we

don’t have to wait. Test automation can

facilitate manual exploration.


Discussion & Questions


Documents

Workshop on Agile Test Strategies and Experiences...‘The nature of the tester's role changes in iterative projects. We are no longer the high-profile victims, we are no longer the