Paul Gerrard
THE
TESTINGOF
Advancing
Testing
Using Axioms
Axioms – a Brief Introduction
Advancing Testing Using Axioms
First Equation of Testing
Test Strategy and Approach
Testing Improvement
A Skills Framework for Testers
Quantum Testing
Close
Formulated as a context-neutral set of rules
for testing systems
They represent the critical thinking processes
required to test any system
There are clear opportunities to advance the
practice of testing using them
Testers Pocketbook: testers-pocketbook.com
Test Axioms Website test-axioms.com
• Test Axioms are not beginners guides
• They can help you to think critically about testing
• They expose flaws in other people‟s thinking and their
arguments about testing
• They generate some useful by-products
• They help you to separate context from values
• Interesting research areas!
• First Equation of Testing, Testing Uncertainty Principle,
Quantum Theory, Relativity, Exclusion Principle...
• You can tell I like physics
Stakeholder, (Test) Design and (Test) Delivery
You have the little book
Stakeholder
Value
Scope Mgt.
Good-Enough
Stakeholderaxioms
Summary:Identify and engage the people or organisations that will use and benefit from the test evidence we are to provide
Consequence if ignored or violated:There will be no mandate or any authority for testing. Reports of passes, fails or enquiries have no audience.
Questions: Who are they?
Whose interests do they represent?
What evidence do they want?
What do they need it for?
When do they want it?
In what format?
How often?
Test Model
Test Basis
Oracle
Coverage
Prioritis-ation
Fallibility
Design axioms
Summary:
Choose test models to derive
tests that are meaningful to
stakeholders. Recognise the
models‟ limitations and the
assumptions that the models
make
Consequence if ignored or
violated:
Tests design will be
meaningless and not credible
to stakeholders.
Questions Are design models available to use as
test models? Are they mandatory?
What test models could be used to derive tests from the Test Basis?
Which test models will be used?
Are test models to be documented or are they purely mental models?
What are the benefits of using these models?
What simplifying assumptions do these models make?
How will these models contribute to the delivery of evidence useful to the acceptance decision makers?
How will these models combine to provide sufficient evidence without excessive duplication?
How will the number of tests derived from models be bounded?
Confidence
Sequenc-ing
Event
Never-Finished
Repeat-Test
Environment
Delivery axioms
Axioms+ Context
+ Values+ Thinking=Approach
Separation of Axioms, context, values and
thinking
Tools, methodologies, certification, maturity
models promote approaches without
reference to your context or values
No thinking is required!
Without a unifying test theory you have no
objective way of assessing these products.
Given context, practitioners can promote
different approaches based on their values
Values are preferences or beliefs
Pre-planned v exploratory
Predefined v custom process
Requirements-driven v goal-based
Standard documentation v face-to-face comms.
Some contexts preclude certain practices
“No best practices”
Separating axioms, context and values clarifies positions, for example:
„Structured‟ (certified?) test advocates have little (useful) to say about Agile contexts
Exploratory test advocates have little (useful) to say about contract/requirements-based acceptance
The disputes between these positions is more about values than practices in context
Is a consultant recommendation best for the stakeholders or the consultant?
Strategy is a thought process
not a document
TestStrategy
Risks
Goals
ConstraintsHuman
resource
EnvironmentTimescales
Process(lack of?)
ContractCulture
Opportunities
User involvement
Automation
De-Duplication
Early Testing
Skills
CommunicationAxioms
Artefacts
1. Test Plan Identifier2. Introduction3. Test Items4. Features to be Tested5. Features not to be
Tested6. Approach7. Item Pass/Fail Criteria8. Suspension Criteria
and Resumption Requirements
9. Test Deliverables10. Testing Tasks11. Environmental Needs12. Responsibilities13. Staffing and Training
Needs14. Schedule15. Risks and
Contingencies16. Approvals
Based on IEEE Standard 829-1998
Used as a strategy checklist
Scarily vague (don‟t go there)
Used as a documentation template/standard
Flexible, not prescriptive, but encourages copy and
edit mentality (documents that no one reads)
But many many testers seek guidance on
What to consider in a test strategy
Communicating their strategy to stakeholders and
project participants
Items 1, 2 – Administration Items 3+4+5 – Scope Management, Prioritisation Item 6 – All the Axioms are relevant Items 7+8 – Good-Enough, Value Item 9 – Stakeholder, Value, Confidence Item 10 – All the Axioms are Relevant Item 11 – Environment Item 12 – Stakeholder Item 13 – All the Axioms are Relevant Item 14 – All the Axioms are Relevant Item 15 – Fallibility, Event Item 16 – Stakeholder Axioms
1. Stakeholder Objectives Stakeholder management
Goal and risk management
Decisions to be made and how (acceptance)
How testing will provide confidence and be assessed
How scope will be determined
2. Design approach Sources of knowledge (bases and
oracles)
Sources of uncertainty
Models to be used for design and coverage
Prioritisation approach
3. Delivery approach Test sequencing policy
Repeat test policies
Environment requirements
Information delivery approach
Incident management approach
Execution and end-game approach
4. Plan (high or low-level) Scope
Tasks
Responsibilities
Schedule
Approvals
Risks and contingencies
Test process improvement is a
waste of time
Actuallyits 11
(most were not software related)
Google search “CMM” – 22,300,000
“CMM Training” – 48,200
“CMM improves quality” – 74 (BUT really 11 – most of these have NOTHING to do with software)
A Gerrard Consulting client… CMM level 3 and proud of it (chaotic, hero culture)
Hired us to assess their overall s/w process and make recommendations (quality, time to deliver is slipping)
40+ recommendations, only 7 adopted – they couldn‟t change
How on earth did they get through the CMM 3 audit?
Using process change to fix cultural or
organisational problems is never going to
work
Improving test in isolation is never going to
work either
Need to look at changing context rather than
values…
Context+ Values
+ Thinking=Approach
<- your values<- your context
<- your thinking<- your approach
Context+ Values
+ Thinking=Approach
<- someone else's<- someone else's
<- someone else's<- someone else's
Axioms+ Context
+ Values+ Thinking=Approach
<- recognise
<- hard to change<- could change?
<- just do some<- your approach
Axioms represent the critical things to think
about
Associated questions act as checklists to:
Assess your current approach
Identify gaps, inconsistencies in current approach
QA your new approach in the future
Axioms represent the WHAT
Your approach specifies HOW
Mission
Coalition
Vision
Communication
Action
Wins
Consolidation
Anchoring
Changes identified here
If you must use one, this is where your ‘process model’ comes into play
Axioms indicate WHAT to think
about...
...so the Axioms point to SKILLS
Summary:
Choose test models to derive tests that are meaningful to stakeholders. Recognise the models‟ limitations and the assumptions that the models make.
Consequence if ignored or violated:
Tests design will be meaningless and not credible to stakeholders.
Questions: Are design models available to use as
test models? Are they mandatory? What test models could be used to
derive tests from the Test Basis? Which test models will be used? Are test models to be documented
or are they purely mental models? What are the benefits of using these
models? What simplifying assumptions do
these models make? How will these models contribute to
the delivery of evidence useful to the acceptance decision makers?
How will these models combine to provide sufficient evidence without excessive duplication?
How will the number of tests derived from models be bounded?
A tester needs to understand:
Test models and how to use them
How to select test models from fallible sources of
knowledge
How to design test models from fallible sources of
knowledge
Significance, authority and precedence of test models
How to use models to communicate
The limitations of test models
Familiarity with common models
Is this all that current certification provides?
Intellectual skills and capabilities are more important than the clerical skills
Need to re-focus on:
Testing thought processes (Axioms)
Testing Stakeholder relationship management
Testing as an information provision service
Goal and risk-based testing
Real-world examples, not theory
Practical, hands-on, real-world training, exercises and coaching.
If evidence arrives in discrete
quanta...
...can we assign a value to it?
As tests are run, every individual test has
some significance
Some tests expose failures but ultimately we
want all tests to PASS
When all tests pass – the stakeholders are
happy, aren‟t they?
Can we measure confidence by counting test?
Not really...
Coverage model: A test could cover one or hundreds of functional conditions, ten thousand
program statements or ten Objective:
Criticality of the business goal it examples
Criticality of the risk it informs Precedent:
The first end-to-end test pass is significant
The 100th variation of a similar test is less significant Functional dependence:
A test of shared functionality used thousands of times per hour could be much more important than a peripheral feature used once/day
Stakeholder: Are customers tests more or less significant than supplier tests?
Context: The same test run at different times in different environments can have
different value.
Only a stakeholder can assign a value to a test (but
that is very hard thing to do)
A tester cannot quantify value, but can define its
significance
A test is significant (to stakeholders) if it: Can be related to a meaningful test objective
Increases coverage with respect to a meaningful test model
Is considered in an acceptance decision (at any level)
Significance is a Boolean but could be 0 or 1
The number of insignificant tests should be zero.
Significance can only be assessed by testers if:
Our test objectives, models, coverage goals are meaningful (to stakeholders) or
Testers are authorised to create their own objectives, measures and coverage goals or
Testers are their own stakeholder
Testers need a close, trusting relationship with their stakeholders or authorised autonomy
E.g. exploratory testing won‟t work if stakeholders do not allow autonomy
Testers should not „go it alone‟.
Test coverage models and goals that generate uniform distributions of tests are inefficient and uninformative
We need more and better test models
Models that are meaningful in context
Significance varies with context and can be used to explain why
e.g. some tests aren‟t useful as regression tests
How much testing is enough?
Can never be answered by coverage alone.
Axioms are context-neutral rules for testing The Equation of Testing
Separates axioms, context, values and thinking
We can have sensible conversations about process
Axioms and associated questions provide context neutral checklists for test strategy, assessment/improvement and skills
Quantum Testing separates significance from value; can it answer the question, “how much testing is enough?”
Thank-You!
THE
TESTINGOF
testaxioms.com
testers-pocketbook.com
gerrardconsulting.comuktmf.com