Paul Gerrard

THE

TESTINGOF

Advancing

Testing

Using Axioms

Axioms – a Brief Introduction

Advancing Testing Using Axioms

First Equation of Testing

Test Strategy and Approach

Testing Improvement

A Skills Framework for Testers

Quantum Testing

Close

Formulated as a context-neutral set of rules

for testing systems

They represent the critical thinking processes

required to test any system

There are clear opportunities to advance the

practice of testing using them

Testers Pocketbook: testers-pocketbook.com

Test Axioms Website test-axioms.com

• Test Axioms are not beginners guides

• They can help you to think critically about testing

• They expose flaws in other people‟s thinking and their

arguments about testing

• They generate some useful by-products

• They help you to separate context from values

• Interesting research areas!

• First Equation of Testing, Testing Uncertainty Principle,

Quantum Theory, Relativity, Exclusion Principle...

• You can tell I like physics

Stakeholder, (Test) Design and (Test) Delivery

You have the little book

Stakeholder

Value

Scope Mgt.

Good-Enough

Stakeholderaxioms

Summary:Identify and engage the people or organisations that will use and benefit from the test evidence we are to provide

Consequence if ignored or violated:There will be no mandate or any authority for testing. Reports of passes, fails or enquiries have no audience.

Questions: Who are they?

Whose interests do they represent?

What evidence do they want?

What do they need it for?

When do they want it?

In what format?

How often?

Test Model

Test Basis

Oracle

Coverage

Prioritis-ation

Fallibility

Design axioms

Summary:

Choose test models to derive

tests that are meaningful to

stakeholders. Recognise the

models‟ limitations and the

assumptions that the models

make

Consequence if ignored or

violated:

Tests design will be

meaningless and not credible

to stakeholders.

Questions Are design models available to use as

test models? Are they mandatory?

What test models could be used to derive tests from the Test Basis?

Which test models will be used?

Are test models to be documented or are they purely mental models?

What are the benefits of using these models?

What simplifying assumptions do these models make?

How will these models contribute to the delivery of evidence useful to the acceptance decision makers?

How will these models combine to provide sufficient evidence without excessive duplication?

How will the number of tests derived from models be bounded?

Confidence

Sequenc-ing

Event

Never-Finished

Repeat-Test

Environment

Delivery axioms

Axioms+ Context

+ Values+ Thinking=Approach

Separation of Axioms, context, values and

thinking

Tools, methodologies, certification, maturity

models promote approaches without

reference to your context or values

No thinking is required!

Without a unifying test theory you have no

objective way of assessing these products.

Given context, practitioners can promote

different approaches based on their values

Values are preferences or beliefs

Pre-planned v exploratory

Predefined v custom process

Requirements-driven v goal-based

Standard documentation v face-to-face comms.

Some contexts preclude certain practices

“No best practices”

Separating axioms, context and values clarifies positions, for example:

„Structured‟ (certified?) test advocates have little (useful) to say about Agile contexts

Exploratory test advocates have little (useful) to say about contract/requirements-based acceptance

The disputes between these positions is more about values than practices in context

Is a consultant recommendation best for the stakeholders or the consultant?

Strategy is a thought process

not a document

TestStrategy

Risks

Goals

ConstraintsHuman

resource

EnvironmentTimescales

Process(lack of?)

ContractCulture

Opportunities

User involvement

Automation

De-Duplication

Early Testing

Skills

CommunicationAxioms

Artefacts

1. Test Plan Identifier2. Introduction3. Test Items4. Features to be Tested5. Features not to be

Tested6. Approach7. Item Pass/Fail Criteria8. Suspension Criteria

and Resumption Requirements

9. Test Deliverables10. Testing Tasks11. Environmental Needs12. Responsibilities13. Staffing and Training

Needs14. Schedule15. Risks and

Contingencies16. Approvals

Based on IEEE Standard 829-1998

Used as a strategy checklist

Scarily vague (don‟t go there)

Used as a documentation template/standard

Flexible, not prescriptive, but encourages copy and

edit mentality (documents that no one reads)

But many many testers seek guidance on

What to consider in a test strategy

Communicating their strategy to stakeholders and

project participants

Items 1, 2 – Administration Items 3+4+5 – Scope Management, Prioritisation Item 6 – All the Axioms are relevant Items 7+8 – Good-Enough, Value Item 9 – Stakeholder, Value, Confidence Item 10 – All the Axioms are Relevant Item 11 – Environment Item 12 – Stakeholder Item 13 – All the Axioms are Relevant Item 14 – All the Axioms are Relevant Item 15 – Fallibility, Event Item 16 – Stakeholder Axioms

1. Stakeholder Objectives Stakeholder management

Goal and risk management

Decisions to be made and how (acceptance)

How testing will provide confidence and be assessed

How scope will be determined

2. Design approach Sources of knowledge (bases and

oracles)

Sources of uncertainty

Models to be used for design and coverage

Prioritisation approach

3. Delivery approach Test sequencing policy

Repeat test policies

Environment requirements

Information delivery approach

Incident management approach

Execution and end-game approach

4. Plan (high or low-level) Scope

Tasks

Responsibilities

Schedule

Approvals

Risks and contingencies

Test process improvement is a

waste of time

Actuallyits 11

(most were not software related)

Google search “CMM” – 22,300,000

“CMM Training” – 48,200

“CMM improves quality” – 74 (BUT really 11 – most of these have NOTHING to do with software)

A Gerrard Consulting client… CMM level 3 and proud of it (chaotic, hero culture)

Hired us to assess their overall s/w process and make recommendations (quality, time to deliver is slipping)

40+ recommendations, only 7 adopted – they couldn‟t change

How on earth did they get through the CMM 3 audit?

Using process change to fix cultural or

organisational problems is never going to

work

Improving test in isolation is never going to

work either

Need to look at changing context rather than

values…

Context+ Values

+ Thinking=Approach

<- your values<- your context

<- your thinking<- your approach

Context+ Values

+ Thinking=Approach

<- someone else's<- someone else's

<- someone else's<- someone else's

Axioms+ Context

+ Values+ Thinking=Approach

<- recognise

<- hard to change<- could change?

<- just do some<- your approach

Axioms represent the critical things to think

about

Associated questions act as checklists to:

Assess your current approach

Identify gaps, inconsistencies in current approach

QA your new approach in the future

Axioms represent the WHAT

Your approach specifies HOW

Mission

Coalition

Vision

Communication

Action

Wins

Consolidation

Anchoring

Changes identified here

If you must use one, this is where your ‘process model’ comes into play

Axioms indicate WHAT to think

about...

...so the Axioms point to SKILLS

Summary:

Choose test models to derive tests that are meaningful to stakeholders. Recognise the models‟ limitations and the assumptions that the models make.

Consequence if ignored or violated:

Tests design will be meaningless and not credible to stakeholders.

Questions: Are design models available to use as

test models? Are they mandatory? What test models could be used to

derive tests from the Test Basis? Which test models will be used? Are test models to be documented

or are they purely mental models? What are the benefits of using these

models? What simplifying assumptions do

these models make? How will these models contribute to

the delivery of evidence useful to the acceptance decision makers?

How will these models combine to provide sufficient evidence without excessive duplication?

How will the number of tests derived from models be bounded?

A tester needs to understand:

Test models and how to use them

How to select test models from fallible sources of

knowledge

How to design test models from fallible sources of

knowledge

Significance, authority and precedence of test models

How to use models to communicate

The limitations of test models

Familiarity with common models

Is this all that current certification provides?

Intellectual skills and capabilities are more important than the clerical skills

Need to re-focus on:

Testing thought processes (Axioms)

Testing Stakeholder relationship management

Testing as an information provision service

Goal and risk-based testing

Real-world examples, not theory

Practical, hands-on, real-world training, exercises and coaching.

If evidence arrives in discrete

quanta...

...can we assign a value to it?

As tests are run, every individual test has

some significance

Some tests expose failures but ultimately we

want all tests to PASS

When all tests pass – the stakeholders are

happy, aren‟t they?

Can we measure confidence by counting test?

Not really...

Coverage model: A test could cover one or hundreds of functional conditions, ten thousand

program statements or ten Objective:

Criticality of the business goal it examples

Criticality of the risk it informs Precedent:

The first end-to-end test pass is significant

The 100th variation of a similar test is less significant Functional dependence:

A test of shared functionality used thousands of times per hour could be much more important than a peripheral feature used once/day

Stakeholder: Are customers tests more or less significant than supplier tests?

Context: The same test run at different times in different environments can have

different value.

Only a stakeholder can assign a value to a test (but

that is very hard thing to do)

A tester cannot quantify value, but can define its

significance

A test is significant (to stakeholders) if it: Can be related to a meaningful test objective

Increases coverage with respect to a meaningful test model

Is considered in an acceptance decision (at any level)

Significance is a Boolean but could be 0 or 1

The number of insignificant tests should be zero.

Significance can only be assessed by testers if:

Our test objectives, models, coverage goals are meaningful (to stakeholders) or

Testers are authorised to create their own objectives, measures and coverage goals or

Testers are their own stakeholder

Testers need a close, trusting relationship with their stakeholders or authorised autonomy

E.g. exploratory testing won‟t work if stakeholders do not allow autonomy

Testers should not „go it alone‟.

Test coverage models and goals that generate uniform distributions of tests are inefficient and uninformative

We need more and better test models

Models that are meaningful in context

Significance varies with context and can be used to explain why

e.g. some tests aren‟t useful as regression tests

How much testing is enough?

Can never be answered by coverage alone.

Axioms are context-neutral rules for testing The Equation of Testing

Separates axioms, context, values and thinking

We can have sensible conversations about process

Axioms and associated questions provide context neutral checklists for test strategy, assessment/improvement and skills

Quantum Testing separates significance from value; can it answer the question, “how much testing is enough?”

Thank-You!

THE

TESTINGOF

testaxioms.com

testers-pocketbook.com

gerrardconsulting.comuktmf.com