A New Model for Testing @paul_gerrard Paul Gerrard paul@gerrardconsulting.com gerrardconsulting.com Programme Chair 2014

Helping clients transform their testing through INNOVATION, COACHING and LEADERSHIP Our CLIENTS Want to be agile rather than follow Agile dogma Have a pragmatic approach and are focused on delivery Want a solution that fits, not a badly fitting suit.

Typical tester job spec? (May 2014) Skills & Requirements Thorough C# knowledge including Linq Good .Net experience including charting and winforms applications. Experience building and maintaining automated test systems Ability to interact with the developers to anticipate future needs Ability to influence to ensure good test procedures are developed and implemented Desirable Skills Some knowledge of Java for mobile testing An interest in photography.

Agenda Pressure to Change Three Innovations Where is this Leading? A New Model forTesting Consequences Shift-Left is More Important than Agile

What's happening? Stampede to mobile computing Big Data Internet of (Every)Thing, Pervasive Computing Continuous Delivery, DevOps (Test) Analytics, Data-Driven Development Shift-left, embedded testers, no test team Did you really think Agile was the last word?

Three Innovations That Are Driving Change Being agile Shift Left Analytics

#1 Agile doesnt work but being agile might Agile is no longer innovative. Challenging Agile and being agile might be Is continuous delivery for you?

Agile doesnt work? It works for some But many organisations have difficulties Mostly a cultural problem Requirements take months to elicit and agree Complex stakeholder relationships Stakeholders with all the knowledge cant be spared to be the on-site customer/product owner Lack of confidence => big system/acceptance tests We are Agile but ...

Pragmatism rules (again) Pragmatic companies use the things that work and discard those that dont Let's be honest: Most are doing some form of hybrid approach: Chaotic interfaces between Agile teams and structured businesses TDD, BDD,ATDD, SBE today Continuous Delivery, DevOps tomorrow?

#2 Shift Left Developers in test? Testers in development? Testing (testers?) everywhere

TestingBDD/Test-Driven) Trusted Requirements + Stories Shift left (redistributed testing) Functional Requirements Build (+little testing) but lots of rework Testing (lots of checking + bugs) Trusted Requirements + Stories Testing SAVED TIME BDD/Test-Driven Who is going to do all this? BAs, Developers or Testers?

From traditional delivery to Continuous Delivery

Deployment into Production is not the Same as Going Live Release to production and test environments on-demand Go Live weekly/monthly/quarterly

#3 Test Analytics Analytics from production Analytics from test An opportunity for testing

Analysis Insight Decision Development and Testing Production Testing IntroducingTest Analytics IntroducingTest Analytics, Paul Gerrard: http://gerrardconsulting.com/index.php?q=node/630 Julian Harty, "Symbiotic Relationships betweenTesting and SMAC": http://blog.bettersoftwaretesting.com/2013/07/symbiotic-relationships-between- testing-and-social-mobile-analytics-cloud-computing/ The capture, integration and analysis of test and production monitoring data to inform business and software development decision-making.

Data silos for test analytics (an illustration) Stakeholder Requirements Assurance (Production) Application Monitoring Production Environment Monitoring Stakeholders Business Goals and measures Stakeholder Involvement/ Engagement Risk Requirements Stories/Feature Descriptions Glossary of terms andTerm Usage Processes Process Paths (Workflows) ManualTests GeneratedTest Code (unit, integration, system level) Application Instrumentation AutomatedTest Execution History Test Status Feature Code Change History Application Process Flows User Accesses/ Activities Feature Calls, Response times Feature use cases (data usage) Application Alerts/Failures Database Accesses Database Content Production failures System assets Resource Usage Log Performance data System Events System Alerts/ Failures/Incidents Outages

Where is this leading? If I knew that, I wouldn't be here...

Three development patterns Structured Agile Continuous Autonomous

There aren't three patterns There are many

From high process to DIY process In the past, high process was seen to be the technical solution to the software problem But technical solutions cannot solve human problems Agile promised flexibility, but a lot of Agile is really 'high process-in-the-small' True agility means a team can adapt or select its process on-the-fly.

The old ways won't work in the future We need a New Model ofTesting (free from logistics)

A New Model for Testing Don't throw stones.Yet. I'm working on a refined version (with others). Want to help/review?

Forget Logistics (for the time being) Document or not? Agile v waterfall? Planned v exploratory? This technology v that technology?

ALLTesting is Exploratory We explore sources of knowledge ... ... to build test models ... ... that inform our testing.

AllTesting is Based on Models

Examples of models

Models are innate, essential, human

Judgement, exploring and testing Testing (the system) Our model(s) are adequate Our model(s) are not adequate Exploring (sources) Judgement Creates test models Uses test models We explore sources of knowledge to build test models that inform our testing BTW Do Developers explore the same way? I think so.

Exploration process ExplorationDefinitions specs/stories Oracles Sources Design Models Test Models Enquiring Challenging Sources: People, documents, experience, system under test Modelling Test Models: Can be documented or mental models Predicting System under test

Testing process Testing System Under Test Refining Informing Applying Interpreting Test Models Revise the System More exploring Reporting

New Model Testing 29 page paper: http://dev.sp.qa/download/newModel

Some Consequences

Do developers and testers think the same way? Similar processes, not thinking the same Easier to embed testers in development Developer testing will improve

Covers functional and non-functional testing? Ignoring logistics means we can have the same foundational process for ALL testing (I think)

Relation to TDD and BDD? TDD is not testing BDD is modelling using stories

Testing v Checking? I submit that a Check is "a test that can be applied by a human or tool" But only humans can interpret

Test automation from a different perspective Automation uses different test models Is that why automation efforts fail so often?

DeveloperTesting The model works (for me)

Capabilities Enquiring, Modelling, Predicting, Challenging Informing,Applying, Interpreting, Refining Reporting and Logging

Analysis, enquiry and elicitation Modelling Creation of custom models, using heuristics, guesses, brainstorming, ideation, creative thinking Custom test design techniques Comparison of models, value, advantages, disadvantages, compromises Identification, validation and use of oracles Predicate logic and proof Hypothesis and inference Socratic method Rapid Review and Inspection techniques Test case design Test models and the meaning of coverage Testing as controlled experiment Observation, Note taking, recording A very different skillset Basic data analysis and statistics Decision-making with incomplete data Computer forensics Fault tree analysis Failure diagnosis Bug advocacy, triage processes and negotiation Meaningful software and test metrics Visual presentation of data Reporting and presentation skills Understanding stakeholders Test analytics Risk management, risk-based testing and decision-making Critical Thinking Interpersonal skills Dealing with uncertainty/fallibility

Shift-Left is More Important than Agile

Developers write code; testers test because developers cant No they dont

Goal: End Manual Feature Checking Let the tools do it and free people to do testing please!

Testing doesnt just apply to existing software, at the end We test systems, not software. We test early, we test often but it might not be called testing

Testers Need to Learn How to Code Some, perhaps many will have to

Testing is about measuring achievement, not quality Align systems delivery with project goals and risks and youve got a seat on the board

Testers dont own testing anymore Be aTestMaster "Assurance through the lifecycle"

New Model Testing changes what testers & developers need to learn Roles of developers/testers become blurred Perhaps they are not different disciplines after all? Why have a tester certification?

The phase after development is REWORK, not Testing Testing must align with development, not compete with it or rescue it

A New Model for Testing http://dev.sp.qa/download/newModel @paul_gerrard Paul Gerrard paul@gerrardconsulting.com gerrardconsulting.com Programme Chair 2014