50
A New Model for Testing @ p a u l _ g e r r a r d Paul Gerrard [email protected] gerrardconsulting.com Programme Chair 2014

A New Model For Testing

Embed Size (px)

DESCRIPTION

I believe that our existing models of testing are not fit for purpose – they are inconsistent, controversial, partial, proprietary and stuck in the past. They are not going to support us in the rapidly emerging technologies and approaches. The certification schemes that should represent the interests and integrity of our profession don’t, and we are left with schemes that are popular, but have low value, lower esteem and attract harsh criticism. My goal in proposing the New Model is to stimulate new thinking in this area. eurostarconferences.com testhuddle.com

Citation preview

Page 1: A New Model For Testing

A New Model for Testing@

paul_gerrard

Paul [email protected]

gerrardconsulting.com

Programme Chair 2014

Page 2: A New Model For Testing

Helping clients transform their testing through

INNOVATION, COACHING and LEADERSHIP

Our CLIENTS– Want to be agile rather than follow Agile

dogma– Have a pragmatic approach and are focused on

delivery– Want a solution that fits, not a badly fitting

suit.

Page 3: A New Model For Testing

Typical tester job spec? (May 2014)• Skills & Requirements

– Thorough C# knowledge including Linq– Good .Net experience including charting and winforms

applications.– Experience building and maintaining automated test

systems– Ability to interact with the developers to anticipate future

needs– Ability to influence to ensure good test procedures are

developed and implemented• Desirable Skills

– Some knowledge of Java for mobile testing– An interest in photography. Wearable Computing

(Internet o

f Eve

rything)

Page 4: A New Model For Testing

Agenda• Pressure to Change• Three Innovations• Where is this Leading?• A New Model for Testing• Consequences• Shift-Left is More Important than

Agile

Page 5: A New Model For Testing

What's happening?• Stampede to mobile computing• Big Data• Internet of (Every)Thing, Pervasive

Computing• Continuous Delivery, DevOps• (Test) Analytics, Data-Driven

Development• Shift-left, embedded testers, no test

team• Did you really think Agile was the last

word?

Page 6: A New Model For Testing

Three Innovations That Are Driving

ChangeBeing agileShift LeftAnalytics

Page 7: A New Model For Testing

#1Agile doesn’t work

but being agile might

Agile is no longer innovative.Challenging Agile and being

agile might beIs continuous delivery for

you?

Page 8: A New Model For Testing

Agile doesn’t work?• It works for some• But many organisations have difficulties• Mostly a cultural problem– Requirements take months to elicit and agree– Complex stakeholder relationships– Stakeholders with ‘all the knowledge’ can’t be

spared to be the on-site customer/product owner– Lack of confidence => big system/acceptance

tests• We are Agile but ...

Page 9: A New Model For Testing

Pragmatism rules (again)• Pragmatic companies use the things

that work and discard those that don’t• Let's be honest: Most are doing some

form of hybrid approach:– Chaotic interfaces between Agile teams

and structured businesses– TDD, BDD, ATDD, SBE today…– Continuous Delivery, DevOps… tomorrow?

Page 10: A New Model For Testing

#2Shift Left

Developers in test?Testers in development?

Testing (testers?) everywhere

Page 11: A New Model For Testing

TestingBDD/Test-Driven)Trusted

Requirements + Stories

Shift left (redistributed testing)

Functional Requirements

Build (+little testing) but lots of rework

Testing (lots of checking + bugs)

Trusted Requirements +

StoriesTesting

SAVEDTIME

BDD/Test-Driven

Who is going to do all this?BAs, Developers or Testers?

Page 12: A New Model For Testing

From traditional delivery…

… to Continuous Delivery

Page 13: A New Model For Testing

Deployment into Production is not

the Same as ‘Going Live’

Release to production and test environments on-demand

Go Live weekly/monthly/quarterly…

Page 14: A New Model For Testing

#3Test Analytics

Analytics from productionAnalytics from test

An opportunity for testing

Page 15: A New Model For Testing

Analysis Insight Decision Development and Testing

ProductionTesting

Introducing Test Analytics

“Introducing Test Analytics”, Paul Gerrard:http://gerrardconsulting.com/index.php?q=node/630Julian Harty, "Symbiotic Relationships between Testing and SMAC":http://blog.bettersoftwaretesting.com/2013/07/symbiotic-relationships-between-testing-and-social-mobile-analytics-cloud-computing/

“The capture, integration and analysis of test and production monitoring data to inform business and software development decision-making”.

Page 16: A New Model For Testing

Data silos for test analytics(an illustration)Stakeholder Requirements Assurance (Production)

ApplicationMonitoring

Production Environment Monitoring

Stakeholders Business Goals

and measures Stakeholder

Involvement/ Engagement

Risk

Requirements Stories/Feature

Descriptions Glossary of

terms and Term Usage

Processes Process Paths

(Workflows)

Manual Tests Generated Test

Code (unit, integration, system level)

Application Instrumentation

Automated Test Execution History

Test Status Feature Code

Change History

Application Process Flows

User Accesses/ Activities

Feature Calls, Response times

Feature use cases (data usage)

Application Alerts/Failures

Database Accesses

Database Content

Production failures

System assets Resource

Usage Log Performance

data System Events System Alerts/

Failures/Incidents

Outages

Page 17: A New Model For Testing

Where is this leading?

If I knew that, I wouldn't be here...

Page 18: A New Model For Testing

Three development patterns

Structured

Agile Continuous

Goal-B

ased

Hi-Process

Autonomous

Page 19: A New Model For Testing

There aren't three patterns

There are many

Page 20: A New Model For Testing

From high process to DIY process• In the past, high process was seen to be

the technical solution to the software problem

• But technical solutions cannot solve human problems

• Agile promised flexibility, but a lot of Agile is really 'high process-in-the-small'

• True agility means a team can adapt or select its process on-the-fly.

Page 21: A New Model For Testing

The old ways won't work in the future

We need a New Model of Testing (free from logistics)

Page 22: A New Model For Testing

A New Model for Testing

Don't throw stones. Yet.I'm working on a refined version (with

others).Want to help/review?

Page 23: A New Model For Testing

Forget Logistics(for the time being)

Document or not?Agile v waterfall?

Planned v exploratory?This technology v that

technology?

Page 24: A New Model For Testing

ALL Testing is Exploratory

We explore sources of knowledge ...

... to build test models ...... that inform our testing.

Page 25: A New Model For Testing

All Testing is Based on Models

Page 26: A New Model For Testing

Examples of models

Page 27: A New Model For Testing

Models are innate, essential, human

Page 28: A New Model For Testing

Judgement, exploring and testing

Testing(the system)

Our model(s) are adequate

Our model(s) are not adequate

Exploring (sources) Judgement

Creates testmodels

Uses testmodels

We explore sources of knowledge to build test models that inform our testing

BTW – Do Developers explore the same way? I think so.

Page 29: A New Model For Testing

Exploration process

ExplorationDefinitions specs/stories

Oracles

Sources

DesignModels

TestModels

Enquiring

Challenging

Sources:People, documents,experience, system under test

Modelling

Test Models:Can be documented

or mental models

Predicting

System under test

Page 30: A New Model For Testing

Testing process

TestingSystem

Under Test

Refining

Informing Applying

Interpreting

TestModels

Revise the System

Logging Revis

ing

More exploring Reporting

Page 31: A New Model For Testing

New Model Testing

29 page paper: http://dev.sp.qa/download/newModel

Page 32: A New Model For Testing

Some Consequences

Page 33: A New Model For Testing

Do developers and testers think the

same way?Similar processes, not thinking

the sameEasier to embed testers in

developmentDeveloper testing will improve

Page 34: A New Model For Testing

Covers functional and non-functional

testing?Ignoring logistics means we

can have the same foundational process for ALL

testing(I think)

Page 35: A New Model For Testing

Relation toTDD and BDD?

TDD is not testing BDD is modelling using stories

Page 36: A New Model For Testing

Testing v Checking?

I submit that a Check is "a test that can be applied by a

human or tool"But only humans can interpret

Page 37: A New Model For Testing

Test automation from a different

perspectiveAutomation uses different test

modelsIs that why automation efforts fail so

often?

Page 38: A New Model For Testing

Developer Testing

The model works (for me)

Page 39: A New Model For Testing

Capabilities

Enquiring, Modelling, Predicting, Challenging

Informing, Applying, Interpreting, Refining

Reporting and Logging

Page 40: A New Model For Testing

• Analysis, enquiry and elicitation• Modelling• Creation of custom models, using

heuristics, guesses, brainstorming, ideation, creative thinking

• Custom test design techniques• Comparison of models, value,

advantages, disadvantages, compromises

• Identification, validation and use of oracles

• Predicate logic and proof• Hypothesis and inference• Socratic method• Rapid Review and Inspection

techniques• Test case design• Test models and the meaning of

coverage• Testing as controlled experiment• Observation, Note taking,

recording

A very different skillset• Basic data analysis and statistics• Decision-making with

incomplete data• Computer forensics• Fault tree analysis• Failure diagnosis• Bug advocacy, triage processes

and negotiation• Meaningful software and test

metrics• Visual presentation of data• Reporting and presentation skills• Understanding stakeholders• Test analytics• Risk management, risk-based

testing and decision-making• Critical Thinking• Interpersonal skills• Dealing with

uncertainty/fallibility

Page 41: A New Model For Testing

Shift-Left is More Important than

Agile

Page 42: A New Model For Testing

Developers write code; testers test

because developers can’t

No they don’t

Page 43: A New Model For Testing

Goal: End Manual Feature Checking

Let the tools do it andfree people to do testing

please!

Page 44: A New Model For Testing

Testing doesn’t just apply to existing software, at the

endWe test systems, not

software.We test early, we test often but it might not be called

testing

Page 45: A New Model For Testing

Testers Need to Learn How to Code

Some, perhaps many will have to

Page 46: A New Model For Testing

Testing is about measuring

achievement, not quality

Align systems delivery with project goals and risks and you’ve got a seat on the

board

Page 47: A New Model For Testing

Testers don’t own testing anymore

Be a TestMaster"Assurance through the

lifecycle"

Page 48: A New Model For Testing

New Model Testing changes what

testers & developers need to

learnRoles of developers/testers become

blurredPerhaps they are not different disciplines

after all?Why have a tester certification?

Page 49: A New Model For Testing

The phase after development is REWORK, not

TestingTesting must align with

development, not compete with it or rescue it

Page 50: A New Model For Testing

A New Model for Testinghttp://dev.sp.qa/download/newModel

@paul_gerrard

Paul [email protected]

gerrardconsulting.com

Programme Chair 2014