"Thinking Strategically About Testing" with Fiona Charles

Welcome to the 2013

This webinar is due to start at 1:30PM. Make sure you s4ck around at the end for the Q&A session and con4nue the conversa4on with the speaker on Twi*er a<er the show!

www.eurostarconferences.com

@esconfs #esconfs @speakerhandle

This webinar is presented by Fiona Charles and she will discuss:

‘Thinking Strategically About Tes>ng’

Join the conversa4on on TwiCer!

Thinking Strategically about Testing

Fiona Charles EuroSTAR Virtual Conference

September 17, 2013

Thinking Strategically about Testing 3 © Fiona Charles 2013

EuroSTAR Online Software Testing Summit, Sept. 2013 @esconfs #esconfs @FionaCCharles



“The terms tactic and strategy are often confused: tactics are the actual means used to gain an objective, while strategy is the overall campaign plan, which may involve complex operational patterns, activity, and decision-‐making that lead to tactical execution.”

Wikipedia entry on “Tactic”



A test strategy is a set of big-‐picture ideas embodying the overarching design of a test campaign

Not a detailed plan Not a set of tactical operations



The design behind the

plan



Your solution to the problem

How to uncover the most important information about the system

Most efficiently & effectively

Within the constraints

With the resources available to you

While managing the risks to your testing







What am I dealing with and how big is it? (=scope)

What limits are there to what I can do? (=constraints)

What is “the most important information” in this context?

What are the risks (and known issues)?



Scope



Anything that affects the size of the testing problem:

System(s), components, features

Browsers, devices

Geographical areas, LoBs

Types of testing…



Constraints



Anything that could limit your testing:

Time

People

Environments

Hardware or software

Organisational rules …



Constraints typically have associated risks



Information that matters



How do you know what that means in this context?



What does quality mean for this system?

In practical terms that drive what we look for and how we test



“Quality is value to some person or persons.”

Gerald M. Weinberg



Value is in the eye…



…of the beholder



Information that matters conveys something significant about system value:

Confirmation of documented requirements

Bugs that impair, impede or threaten value



Value(s)

Business drivers for the project

Benefits that stakeholders expect this project to deliver with/in this system

Overriding facts, principles, and beliefs



This system will be used to mobilise emergency services, and guide and inform incident responses based on complex combinations of configurable rules. Incident history will be recorded and tracked in the system by dispatchers and response team leads from receipt of the call through to closure of the incident.



The system is strategic to the business. Threats to the publicly advertised launch date will be catastrophic to the company’s reputation and share price (and, not incidentally, to the executives’ compensation.



The product is a web site that will go live on a specific date to be hit by many thousands of users in an event-‐based marketing campaign. If it crashes or crawls, your company’s name will be mud and it will lose its biggest, most prestigious customer.



The system implements eHealth records that will be accessed and updated nationally by doctors’ offices, private clinics and public hospitals. Patients will be able to access their own records via mobile devices.



“…to some person or persons”



Potential stakeholders for testing

Anyone who could be impacted by implementation of this system:

Expects to benefit

Could be victimized by it

Will have to look after it

Could illegitimately benefit (disfavoured stakeholder; negative value)



Business stakeholders to consider Management—those who pay for the system and therefore have a “stake” in its successful operation

Product owner or manager

Hands-‐on system users

Users of system outputs Reports Data

Customers…



Silent or absent stakeholders who could be significantly impacted

Anonymous customers

Your business customer’s customers

Bystanders



Technical and IT stakeholders to consider

Computer operations (batch jobs, backups, etc.)

Maintenance leads

Help desk…



Of these, whose interests are most significant to include or represent in your testing?



Risks



Testing risks operate at 2 principal levels important for strategy

System/Product risks – The potential for, and impact to stakeholders of, threats to value in the implemented system

Testing project risks – anything that could threaten your ability to solve the testing problem



Initial brainstorming – system risks overall

Are there any “givens” we can start with?

What kinds of impact matter in this organization?

What’s the worst thing that could happen?

Who could be hurt and who do we care about most (hierarchy of concern)?

How could this system hurt each of those?

Which aspects of quality should we look at in this system?



Test project risks can come from anywhere

Scope (too large?)

Constraints

Circumstances (late development)…





Your solution to the testing problem

How to uncover information about threats to stakeholder value

Within the constraints

While managing the risks

With the resources available to you



The core of any test strategy is the model



Even if your test is driven entirely by use cases

Or user stories

Or requirements…



User setup (from Store for Pilot)Store parameters

Item load (from spreadsheet)

1-time setup before all cycles

CENTRAL OFFICE

Item maintenancePrice maintenance

Promotions

CENTRALOFFICE

Once per cycle events

Close store

Verify:Totaller

Cash BalanceDaily Reports

TLOG

STORE1 BACKOFFICE

STORE1 BACKOFFICE

Open store

Close store

Verify:Totaller

Cash BalanceDaily Reports

TLOG

STORE2 BACKOFFICE

STORE2 POS REGISTERS

Open registerLogin

POS transactions...

LogoutClose register

(Verify POS receiptsTill balance)

STORE2 BACKOFFICE

Open store

Daily, 3-4 days per cycle

STORE1 POS REGISTERS

Open registerLogin

POS transactions...

LogoutClose register

(Verify POS receiptsTill balance)

CENTRALOFFICE

Verify:Central reports

Sales report

Once per cycle event

Run reports

STORES

Model for testing a POS system



Why model consciously?

In every test, we make choices What to include What to leave out

Consciously modeling a test gives us a way to control, examine, and explain those choices



Overall model for testing on a large insurance project:



Many possibilities for designing the model that will best help solve your testing problem



Business operations

Business model through a day, week, month, year Product definition & pricing Periodic promotions setup and aging Open store day Sell/return, other transactions Close store day Balance the day Produce financial reports Multiple stores + Central office



Functional or organizational decomposition

Functional areas within the system or business (Ordering, Inventory Management, Billing, etc.) Processes in each area (Order capture, provisioning, etc.) Functions within each process (Enter, edit, cancel order, etc.)



System (or integration) data

Static Semi-‐static Dynamic (see my Modelling Scenarios using Data)



Others

Lifecycle of any entity: A bank account, from open through to close A purchase, from order through extended warranty servicing

Stakeholder experience Users Customers Managers/approvers

State transitions…



It’s a good idea to combine 2 or more models

Each model type can act as a source of test ideas Different models can act as cross-‐checks on each other, generating ideas for complementary scenarios, e.g.,

Combining the real-‐world view of a business operations model with the systems view of a model based on the data



You may need more specific strategies

Priorities

Constraints

Risk management

Resource strategies (data, environments, staffing, tools…)



Scenario ListGL SIT Master (1/cycle/transit)

= TC ID, transaction, session, GLs, Days

TestTeamAnalysis

Transaction/Event

Variations

Transaction/Event

Variations

Transactions& Events

System Test CasesManual

DayPlans

DayPlansDay

Plans

TransactionWorksheet

Transaction/Test CaseWorksheet

Transaction/Test Case

Worksheets

DayTotals

DayTotals

DayTotals

forVerification

Manual

MacroGenerated

DayVerification

Form

Teller ProjectE2E SIT Artifacts

Man

ual

DayPlans

DayPlans

CyclePlans

ProjectMICR Master File



ProjectPartner Files(1/transit/day)

= MICRs as mockup forpaper instruments

Macro

Macro

MacroDataTablesnot used





Transaction/Event

Variations

Transaction/Event

Variations



DayPlans

DayPlansDay

Plans




Worksheets

DayTotals

DayTotals

DayTotals

forVerification

Manual

DayVerification

Form

Teller ProjectE2E SIT Test Artifacts

Man

ual

DayPlans

DayPlans

CyclePlans






Macro

Macro


Posted dailyfor verification





Transaction/Event

Variations

Transaction/Event

Variations



DayPlans

DayPlansDay

Plans




Worksheets

DayTotals

DayTotals

DayTotals

forVerification

Manual

DayVerification

Form

Teller ProjectE2E SIT Test Artifacts

Man

ual

DayPlans

DayPlans

CyclePlans






Macro

Macro


Filed &retained for

Audit

ProblemDatabase

Hard Copy GLReports





Testers love detail!







Work through levels of detail



Thinking tools

Get a big picture

Clarify

Develop ideas

Understand relationships

Explore details

Expose gaps









Ethical conundrums

Practical decisionsHow

Reconcile conflicts

What The "right" thing

What employers expect Explicitly

Ethics policies

Implicitly

What we bring

Strengths Personal value system

Ethical training

Philosophical or religious underpinnings

Self-respect

Self esteem

Undermining influences

Respect for authority Habit of obedience

Fear

Reluctance to stand out or make waves

ObligationsEmployers

Society

Our dependents

Ourselves

A "higher power"Religious

Ethical ideals Our craft/profession

Co-workers

StakeholdersWho are they?

Who is most important?

Why do we believe that?





The representation (documentation) of a thing is not the thing itself



A document (diagram, mindmap…) may describe or express a test strategy but it is not in itself a test strategy



There is no inherent relationship between the rigour of a test effort and the number of pages you take to describe it



Documentation of a test strategy could legitimately be non-‐existent, minimal, or extensive It could be expressed in pictures, words or both



The bigger the document, the lower the likelihood that you will get honest and useful feedback from busy stakeholders



You should be able to summarise your strategy for any test effort—large or small—in no more than a page or two

The model Words Picture(s)



A test strategy document is not made to look “professional” by inclusion of more stuff



Continuous prose with lots of words

Textbook definitions

Information copied from other documents

Tactical details

SOPs (like defect management process)

Generic risks



You can think strategically about testing at any level, for any size of test effort, and at any stage

the politics of testing how to overcome constraints…



I hope to see you at EuroSTAR!

Join me in the Test Lab Tues Nov 5 at 5 pm for the 1-‐hour Test Strategy Challenge

And for my keynote:

“Get in there and argue!” A questioning tester’s personal journey

Fiona Charles fiona.charles@quality-‐intelligence.com

www.quality-‐intelligence.com

Twitter: @FionaCCharles

Images and text ©Fiona Charles 2013



Questions & discussion

Thanks for listening!

Please type your ques>ons for the speaker into the box provided.

@esconfs #esconfs www.eurostarconferences.com

Go to www.eurostarconferences.com for more informa4on

Don’t miss Fiona Charles Keynote Talk: “Get In There And Argue!” A Questioning Tester’s Personal Journey at the 2013 EuroSTAR Conference.

Technology

"Thinking Strategically About Testing" with Fiona Charles