HBT a revolutionary approach to testing software

Hypothesis Based Testing (HBT)A revolutionary approach to testing software

Webinar: 22 June 2011, 1430-1530 IST

T [email protected]

in.linkedin.com/in/AshokSTAG ash_thiru

mailto:[email protected]

mailto:[email protected]

© 2011. STAG Software Private Limited. All rights reserved.

How can we seek inspirations from nature to deliver “Clean Software”?

In the next hour, let us look at the act of testing differently and see how we can think better and test scientifically?


Quality is...

“... at least meeting expectations”

Meeting expectations implies“satisfying properties”

i.e. Expect certain properties to be satisfied.

Properties are impeded by “potential defects”


Properties of matter

http://www.science.wellsresearch.org/wp-content/uploads/2010/02/CBScienceHSPhysicalScienceSt2.jpg

Physical & Chemical properties of matter allow us to:... classify... understand behaviors, interactions... enable checking purity

Think!How can we use a similar train of thought to identify “properties of cleanliness” and then “types of defects”?




“affected by”

“Properties of the system”Cleanliness criteria

Potential Defect Types (PDT)

End user expectations

Issues in specifications,structure, environment

and behavior


Expectations

Needs

Features

Environment

Behavior

Structure

Material

“impedes”

“Properties of the system”Cleanliness criteria

Potential Defect Types (PDT)

Expectations delivered by Needs (Requirements)via Features that display Behaviorconstructed from Materials in accordance to a Structure in a given Environment


Fractional Distillation

From : http://withfriendship.com

A technique to separate mixturesthat have components of different

boiling points

Think!There are a variety of defect types that may be present in the system.

To optimally uncover the defects, can we separate these types of defects on the basis of certain properties and optimally uncover the defects?

http://withfriendship.com/images/d/19767/Fractional-distillation-wallpaper.gif

http://withfriendship.com/images/d/19767/Fractional-distillation-wallpaper.gif


Fetal growth - Delivering systems (Babies too!)

Source :http://www.environment.ucla.edu/media/images/Fetal_dev5.jpg

Visualizing health :- Size, Shape, Parts- Types of issues not present

Think!How can we measure the ‘intrinsic quality’ at a stage?

Quality GrowthAt each stage certain types of defects detected & removed. Quality grows.

Can we chart this as “cleanliness index”?

http://www.environment.ucla.edu/media/images/Fetal_dev5.jpg

http://www.environment.ucla.edu/media/images/Fetal_dev5.jpg


Sherlock Holmes

Focus on intellectual power to solve mysteries.

Strong application of deductive logic.

Think!Can we hypothesize as to what can go wrong and prove/disprove?

Hypothesize potential defect types and prove presence of instances. If they are not present, can we reason their absence?


Test methodologies focus on activities that are driven by a process which are powered by tools, yet successful outcomes still depend a lot on experience.

Typically methodologies are at organizational level.

On the other hand HBT... is a personal scientific methodology enabled by STEMTM ,a defect detection technology to deliver “Clean Software”


Scientific approach to uncovering defects

Cleanliness criteria What is the end user expectation of “Good Quality”?

Potential Defect Types What types of issues can result in poor quality?

Evaluation Stage When should I uncover them?

Test Types How do I uncover them?

What are the test cases? Are they enough?Scenarios/Cases

Test Techniques What techniques to generate test cases?

Scripts How do I execute them?

Metrics & Management How good is it? How am I doing?


HBT: Quality Levels, Aspects & Issues

Input cleanliness

Input interface cleanliness

Structural integrity

Behavior correctness

Environment cleanliness

Attributes met

Robustness

Clean Deployment

End user value

L1

L2

L3

L4

L5

L6

L7

L8

L9

DataInput data handling

Data, LogicFunctionality

StructureInternal structural issues

Structure, UsageUI issues

Logic, EnvironmentError handling

EnvironmentResource leaks, Compatibility...

Structure, Logic, UsagePerformance, security, volume, load...

Environment, Data, LogicCompatibility, migration

UsageUser flows, experience

Aspect Issues


HBT - A Quick Introduction

Understand EXPECTATIONS

Understand CONTEXT

Formulate HYPOTHESIS

Devise PROOF

Tooling SUPPORT

Assess & ANALYSE

S1

S2

S3S4

S5

S6

D1D2

D3D4D5

D6

D7STEM

D8

32 core concepts

D1

Business value understanding

Defecthypothesis

Strategy &planning

Test designTooling

Visibility

Execution & reporting

Analysis &management

D2

D3

D4D5

D6

D7

D8

STEM Core

SIX stages of DOING

EIGHT disciplines of THINKING

powered by

HBTPersonal test methodology

STEMDefect detection technology

powered by


Business value understanding D1Landscaping ViewpointsReductionist principleInteraction matrixOperational profilingAttribute analysisGQM

EFF model (Error-Fault-Failure)Defect centricity principleNegative thinkingOrthogonality principleDefect typing

Defect hypothesisD2

Reductionist principleInput granularity principleBox model Behavior-Stimuli approachTechniques landscapeComplexity assessmentOperational profiling

Test designD4

Orthogonality principleTooling needs assessmentDefect centered ABQuality growth principleTechniques landscapeProcess landscape

Test strategy & planningD3

ToolingD5Automation complexity assessmentMinimal babysitting principleSeparation of concernsTooling needs analysis

GQMQuality quantification model

VisibilityD6

Gating principleCycle scoping

Analysis & ManagementD8

Contextual awarenessDefect rating principle

Execution & ReportingD7

32 core concepts


Activities ........................................................................................................................................................ ............................................................................

Goal

Pow

ered

by

expe

rien

ce

Pow

ered

by

defe

ct

dete

ctio

n te

chno

logy

(S

TEM

)

drives

hopefully results in

Typical

How is HBT different?

Activities ........................................................................................................................................................ ............................................................................

Goal

HBT

from other methodologies...


Complete test cases

Sensible automationGoal directed measures

Staged & purposeful detection

Potential defect typesCleanliness criteria

Expectations

S1

S2S3

S4

S5S6

Hypothesis Based Testing (HBT)


Clear Baseline

Set a clear goal for quality

Example: Clean Water implies1.Colorless2.No suspended particles3.No bacteria4.Odorless

Complete test cases




Expectations

S1, S2

What information(properties) can be used to identify this?

...Marketplace,Customers, End users

...Requirement(flows), Usage, Deployment

... Features, Attributes

...Stage of development, Interactions

... Environment, Architecture

... Behavior, Structure


A goal focused approach to cleanliness

Complete test cases




Expectations

Scientific approach to hypothesizing defects is about looking at

FIVE Aspects - Data, Logic, Structure, Environment & Usage from THREE Views - Error injection, Fault proneness & Failure

Use STEM core concepts > Negative thinking (Aspect)> EFF Model (View)

Identify potential defect types that can impede cleanliness

Example:Data validationTimeoutsResource leakageCalculationStoragePresentationTransactional ...

S3

“A Holmes-ian way of looking at properties of elements”


Levels, Types & Techniques - STRATEGY

Complete test cases




Expectations

Quality Levels

PDT4PDT3

PDT6PDT5

PDT7

PDT1

PDT2 PDT:Potential Defect Types

L1

L2

L3

NINE levels to Cleanliness

Input cleanliness



Behaviour correctness


Attributes met

Robustness

Clean Deployment

End user value

L1

L2

L3

L4

L5

L6

L7

L8

L9

Test Types

PDT2PDT1

PDT4PDT3

PDT6PDT5

PDT7

TT1TT2

TT4

TT5

TT3

TT:Test Types

Test Techniques (T1-T4)

TT1

TT2

TT4

TT5

TT3

T1

T2

T3

T4

S4

“Fractional distillation of bug mixture”


Countable test cases & Fault coverage

Complete test cases




Expectations

That test cases for a given requirement shall have the ability to detect specific types of defects

FAULT COVERAGE

Test Scenarios/Cases

TTTS1 TC1,2,3

TS2 TC4,5,6,7

R1

R2

R3

PDT1

PDT2

PDT3

Requirements & Fault traceability

Use STEM Core concepts> Box model > Behaviour Stimuli approach> Techniques landscape> Coverage evaluation

to - Model behaviour- Create behaviour scenarios- Create stimuli (test cases)

Irrespective of who designs, #scenarios/cases shall be same - COUNTABLE

Guarantee test adequacy.Guarantee implies

that the means to the end is rational & provable

S4


HBT Test Case Architecture

Organized by Quality levels sub-ordered by items (features/modules..), segregated by type, ranked by importance/priority, sub-divided into conformance(+) and robustness(-), classified by early (smoke)/late-stage evaluation, tagged by evaluation frequency, linked by optimal execution order, classified by execution mode (manual/automated)

A well architected set of test cases is like a effective bait that can ‘attract‘ defects in the system.

It is equally important to ensure that they are well organized to enable execution optimisation and have the right set of information to ensure easy automation.

Level

Item

Type

Priority

Focus

Stage

Frequency

Order

Mode


Focused scenarios + Good Automation Architecture

Complete test cases




Expectations Cleanliness criteria

Expectations

Level based test scenarios yield shorter scripts that are more flexible for change and easily maintainable.

Input cleanliness



Behavior correctness


Attributes met

Robustness

Clean Deployment

End user value

L1

L2

L3

L4

L5

L6

L7

L8

L9

S5


“Cleanliness Index” - Improved visibility

Complete test cases




Expectations

Quality reportQuality reportQuality reportQuality reportQuality reportCC1 CC2 CC3 CC4

R1R2R3R4R5

Met

Not met

Partially met

PDT3PDT2

PDT1

TT2

TT1

PDT5PDT4

PDT8

PDT7PDT6

TT5

TT4

TT3L1

L2

L3

L4

Stage

PDT10

PDT9

PDT9 TT6

TT7

TT8

Clea

nlin

essS6

“Growth of a baby”


HBT - A Case Study Two Teams, one using HBT & The other conventional approach

Effort details (person-hours)Effort details (person-hours)Effort details (person-hours)

Stage HBT Normal

Test analysis & Design 30 20

Front loading of effort resulted in lowering support cost

Test case detailsTest case detailsTest case detailsTest case details

Module HBT Normal Increase

M1 100 28 257%M2 85 52 63%M3 95 66 44%M4 132 72 83%M5 127 28 354%M6 855 116 637%

TOTAL 1394 362 285%

Nearly 3x increase in #test cases increasing probability of higher defect yield

Test case details (HBT)Test case details (HBT)Test case details (HBT)Test case details (HBT)

Module Total Positive Negative

M1 100 59 41

M2 85 68 17

M3 95 67 28

M4 132 112 20

M5 127 85 42

M6 855 749 106

TOTAL 1394 1140 254

2x improvement in negative cases increasing probability of better defect yield

Defect detailsDefect detailsDefect detailsDefect details

HBT Normal Increase

#Defects 32 16 100%

20 (Major), 12(Minor)Out of these 32 defects, few were

residual defects, one being critical to corrupt the entire data.


HBT Results

Re-architecting test assets increases test coverage by 250%

50%-1000% reduction in post-release defects

30% defect leakage reduction from early stage

‘Holes’ found & fixed at requirement stage

Test assessment accelerates integration, de-risks deployment

Smart automation - 3x reduction in time

Deskilling - Less experienced staff do better, faster ramp up, lower cost


Summarizing...

Complete test cases




Expectations

S2

S1

S6

S3

S4

S6

Focused scenarios + Good automation architecture

“Cleanliness Index” - Improved visibility

Countable test cases & Fault coverage

Levels, Types & Techniques - STRATEGY

A goal focused approach to cleanliness

Clear baseline


Thank you!

“...the value addition for us came from a blend of HBT/STEM and the dedication/hard work from senior management to excel in their services....” Praveen Kanipakam, President SHARP Software Development India

“..the quality of test design and completeness of testing as a result of their methodology is evident in every release.” Iype Isaac, CTO Bankbazaar.com

“..the focus on scientific way of doing testing with their methodology HBT was reflected in the effectiveness of test artifacts created by their team.” Raj Khare, Chairman & Founder SureWaves

“...the methodology and team was the potent combination that jumpstarted our QA." Tapesh Kumar Agarwal, CTO & Co-Founder Meddiff Technologies

STEMTM is the trademark of STAG Software Private Limited

www.stagsoftware.com

@stagsoft

blog.stagsoftware.com

Connect with us...

http://www.stagsoftware.com

http://www.stagsoftware.com

Documents

HBT a revolutionary approach to testing software