How to Eliminate Manual Feature Checking

How to Eliminate Manual Feature Checking

@paul_gerrard

Paul [email protected]

gerrardconsulting.com

Overview• Checking• Scope of Feature Checking• Why Late, Manual Checking is Bad

for Your Team and Your System• Redistributed Testing• How to Automate Feature Checking• Summary• Q&A

Intelligent Definition and Assurance Slide 2

Checking

Intelligent Definition and Assurance

What is check(ing) versus test(ing)?• There’s an ongoing debate about what

checking is• Here’s my selected definitions of two words:– Test: A procedure for critical evaluation, a means

of determining the presence, quality, or truth of something; a trial

American Heritage DictionaryThis is what I used in the testers-pocketbook.com

– Check: A control to secure accuracy; a test or examination to see that something is correct or in good working order

Oxford Reference Dictionary

Slide 4

http://testers-pocketbook.com/

A check is…• A control to secure accuracy• A comparison with a required outcome (derived

or computed) that tells us we align with some plan or intent

• A test or examination to see that something is correct or in good working order• Correct – compared with some source of

knowledge (a spec, a person, experience, standard or convention)

• In good working order – aligned with someone’s view of how a system ought to work, in some context.Intelligent Definition and Assurance Slide 5

Checking dominates manual testing• By ‘manual’ we mean that the tester thinks

their way through a test, but uses their hands (rather than tools) to drive it

• Most testing is actually checking (regrettably)• Mechanical, procedural, pre-meditated

process• Performed by humans it can be error-prone• Why do we ask human beings to perform an

activity that is (obviously?) better done by tools?


The shift to Agile is irresistible• Agile approaches – BDD/TDD, user stories,

multi-skilled teams, short iterations, rapid feedback, continuous integration, continuous delivery etc. are creeping into all projects

• Automated developer testing (BDD/TDD)• Manual testing at the end is being squeezed• Testers may have to ‘let go’ of manual

checking and embrace redistributed testing• Will automated checking be mandatory in

future?


Scope of Feature Checking

How to identify a feature• A software feature has these

characteristics…–Meets a particular need of a user and/or be

accessible to them (a screen, transaction etc.)– Provide a technical capability and be

implemented as a function, method, class, module, unit

– Can be described in isolation – like a component– Can be tested in isolation– Tests (checks) can be specified in advance.


Scope of feature checking• Behaviour can be derived (from a requirement or

conversations) in advance• A scenario can be defined in a systematic way:

– Pre-conditions (an initial state, setup or preparation)– (Steps) inputs, commands or other stimuli– Post-conditions (outcome(s), outputs, final state…)

• Similar to a traditional test case• Execution and comparison of actual to required

behaviour comprises the check• DeFOSPAM is a method for deriving good scenarios• NB – checks focus on a selective subset of observable

behaviour.


Story HeaderFeature: ship ordersAs a orders clerkI want to acknowledge and ship the order

So that we fulfil a book order

Scenario: ship a single book from stockGiven I select a valid order And the ordered book is in stockWhen I choose ‘acknowledge and ship’Then order status is changed to ‘shipped’ And an address label is printed

Structured stories (other variations exist)

Key wordStory text Each Story has multiple

ScenariosScenarios can be data driven

PRE-CONDITION(S)

STEP(S)

POST-CONDITION(S)

DeFOSPAM – Google itIntelligent Definition and Assurance Slide 11

Success or failure• A feature succeeds (passes the check) if:– The pre-conditions are achieved AND– The system accepts the inputs, commands

or other stimuli AND– The required outcome(s), outputs or final

state are observed• A feature fails (fails the check) if:– The pre-conditions cannot be met, rejects or

does not request the inputs, or the required outcomes are not observed.


Why do checks fail?• The requirement was vague– Developer made bad assumption and

user/tester ‘knows better’– Or the tester made the bad assumption

• System tests more thorough than dev tests?– Greater coverage, more focused, deeper

checking– Risk-based approach– Deeper insight into user behaviour.


Why Late, Manual Checking is Bad for Your Team and Your

System

Rework

Rework

The usual process?


Requirement or story

Initial Development

Initial Testing

Diagnosis, discussion etc.

Initial user Idea

Refine Requirement

Refined requirement

Development Rework

Re-Test

Most rework is due to poor requirements than poor implementation

Wasteful development• We take a stab at the requirement,

believing that users don’t really know what they want

• Developers make assumptions, and so do the testers (sometimes different)

• When software is available we show the users and they say, “that’s great, but what I really want is…”

• How wasteful!Intelligent Definition and Assurance Slide 16

Validating requirements with examples• Wasteful: ask developers to create some code for users to

critique the REQUIREMENT– We use developers to create this ‘straw man’, ‘prototype’, ‘starter

for 10’– The result is a new requirement and some code that doesn’t work

• Instead, why don’t we create some proposed examples of the system in use?– It can encompass the same assumptions, proposals and errors as

the straw man– Our users need just a little more imagination to validate the

requirement• We achieve a refined requirement faster, cheaply, because

we give users concrete examples and don’t ask developers to guess.


Late feature checking by testers is bad• Devs have a safety-net so they test less/test badly• Testers do even more checking – a vicious circle– Testers join teams late, stakeholders not engaged early– Code and tests based on faulty, mistrusted requirements– Poor tests raise non-problems and miss real problems– Bigger test teams, lots of bugs, lots of rework

• Testers shortcut and put checks in long, brittle, unmaintainable end-to-end test scripts– BTW, these tests could be manual OR automated.


Redistributing TestingChecking

If a requirement drives today’s delivery it must be trusted• Organisations demand more rapid and

reliable delivery• Continuous delivery is a hungry beast that

eats requirements• Requirements must be trusted to be

mature, complete and coherent enough to deliver the business value envisaged by stakeholders

TRUSTED, NOT PERFECTIntelligent Definition and Assurance Slide 20

TestingBDD/Test-Driven)Trusted

Requirements + Stories

Redistributed testing (checking)


Functional Requirements

Build (+little testing) but lots of rework

Testing (lots of checking + bugs)

Trusted Requirements +

StoriesTesting

SAVEDTIME

BDD/Test-Driven

How to Automate Feature Checking

Refine

Auto-Checked

Redistributed = story-driven


Initial User Idea

Validate Requirement

Test/Behaviour -Driven

Development

Testing (little checking)

Requirement

Stories

Output: Trusted requirements

Output: Continuously Checked code

Less/no checking, better tests, fewer

failures

Varying levels of feature checking• ‘Key Examples’

– To verify that team understanding of the requirement is correct

• Feature unit tests– To feed test-driven development

• Feature acceptance criteria– To provide evidence so a feature can be accepted

• Feature system tests– To ensure a feature is tested thoroughly

• Integrated, end-to-end, user tests are really important– But they aren’t feature checks.


Key tool requirements


• BDD tools• xUnit• Other test

frameworks

• BSM• Custom

interfaces required

• BDD tools• BSM

• Traditional RM tools don’t use stories

• BSM Integrated Requirements

and Stories

Test code generation

from stories

Test Execution

Automation

Execution feedback and

reporting

BDD tools support the right hand quadrants

Emerging tools support the left hand quadrants

Business Story Platform Robot Framework


Requirements

Story (structured) Test Cases

Test Export

Keyword Definitions

+

Write the Code+ Refactor the

Code

Test Librarye.g. Webdriverexecutes tests

Test StatusRedAmberGreen

Dictionary

System Under Test

Test Outcomes

Business Story Manager

Business Story Manager and Test Frameworks

Requirement

STORY: Feature Header

Business ScenarioGiven “Search page”When “I find a book” And “I add to basket”Then “basket has my book”

Just enough to clarify understanding

Test ScenarioPre-conditions “Search page”Steps “enter search text”

“click search button”“scroll down to find book”“select book”“view more information to

verify”“select add to basket”

Post Conditions/outcomes “Basket has selected book”

Generated

Story BasedTest Cases

Example: Robot FrameworkTestLog

Selenium (Web) HTTP (web) SSH DB Library Etc.

Manually maintainedKeyword

Definitions 1 per Feature

Keyword Definitions

1 per application

Technical CodePer Environment

TestConfig.

Drivers

System(s) Under Test

Generated code

Four execution automation choices

Story Scenario Test Export

xUnit Test code

Test Framework Keywords

GUI Test Tool Script

Maintained code

Application API calls

Keyword Definitions/Library Calls

GUI Script Function

Definitions

BDD Tool Steps Step DefinitionsCurrent BDDTools

ProprietaryExec. Tools

Generated code

Four execution automation goals

Story Scenario Test Export

xUnit Test code

Test Framework Keywords

GUI Test Tool Script

Goal

Developer Testing

System Feature Checking

System Feature Checking

BDD Tool Steps Developer Testing

Business Story Manager: interfaces

for all four paths

Getting started1. Get testers to create (stories and)

scenarios/acceptance criteria - EARLY– They would create test cases anyway

2. Give developers trusted requirements, stories and scenarios

3. Developers to adopt BDD/TDD4. Testers:

a) Adopt Acceptance-Driven approachb) Implement automated checking


Summary• Improved requirements ‘by example’• Easier route for developers to adopt

BDD/TDD• Automatable feature test cases and reduced

demand for feature-level checking in system test

• Fewer testers, or testers released to do more end to end and exploratory testing

• Trusted, ‘live specifications’ for the lifetime of an application.


Survey

References• Our THINKING: Business Story Method

(overall method, DeFOSPAM etc.)– http://businessstorymethod.com– A Unified Approach to System Functional

Testing (1993)– Testing Requirements (1994)

• Our TECHNOLOGY– http://businessstorymanager.com

• Free Story Platform for QA (a subset of BSM)– http://sp.qa

• If you want to know how to implement the Business Story Method or tool – do get in touch


http://businessstorymethod.com/

http://businessstorymethod.com/

http://businessstorymanager.com/

http://sp.qa/

How to Eliminate Manual Feature Checking

@paul_gerrard

Paul [email protected]

gerrardconsulting.com

Documents

How to Eliminate Manual Feature Checking