A practical Test Strategy A practical Test Strategy

Jurian van de LaarJurian van de Laarjla@improveqs.nljla@improveqs.nl

EuroSTAR Conference 2009EuroSTAR Conference 2009StockholmStockholm

Accredited provider ISTQBFoundation & all Advanced modules

Training provider IREBRequirements Engineering

Accredited Lead Assessorsformal TMMi assessments

Improve Quality ServicesImprove Quality Services

Leading in Testing & Quality ManagementLeading in Testing & Quality Management Consultancy, Subcontracting and TrainingConsultancy, Subcontracting and Training Founded in 1998 by Erik van VeenendaalFounded in 1998 by Erik van Veenendaal

Eindhoven

Improve Quality ServicesWaalre (near Eindhoven)

The Netherlands

www.improveqs.nl

Policy? Strategy?Policy? Strategy?

What does it mean ??What does it mean ??

Do you need this ??Do you need this ??

Case study Case study Critical success factorsCritical success factors

Sounds familiar ?Sounds familiar ?

Definitions Test Policy & StrategyDefinitions Test Policy & Strategy

Test Policy Test Policy – High level description of principles, – High level description of principles, approach and major objectives of the organization approach and major objectives of the organization regarding testing.regarding testing.

Test StrategyTest Strategy – High level description of test levels for – High level description of test levels for an organization or programme, and the testing to be an organization or programme, and the testing to be performed within each level.performed within each level.

Test Policy

Test Strategy

Master Test Plan

Generic risksGeneric risksGeneric risksGeneric risks

Business goalsBusiness goalsBusiness goalsBusiness goals Test goalsTest goalsTest goalsTest goals

BlueprintBlueprintBlueprintBlueprint

ProjectsProjectsProjectsProjects

Organization

Test Policy 20XX

Definition of testing

Mission / Ambition based onbusiness goals

based onbusiness goals

EmployeesTest processCustomers

Test Process Improvement- objectives, performance indicators, model

Test targets

Test StrategyTest Strategy

Based on the test policyBased on the test policy

Addresses the generic product risksAddresses the generic product risks

Contains description of test levels:Contains description of test levels:

� Purpose, responsibility� Level of independence (organizational)� Entry and exit criteria� Test process and deliverables � Approach (top-down, bottom-up)

� Test design techniques� Tools to be used� Standards to be used� Test environment� Typical non-functionals

Test policy & strategy are important … Test policy & strategy are important … Provide common basis for projectsProvide common basis for projects

What is What is reallyreally important for business / important for business / managersmanagers

Leading in models like Leading in models like TMMi, TPI, and TMap NextTMMi, TPI, and TMap Next

… … but also difficult to realize !but also difficult to realize ! ‘‘We have no business goals or policy’We have no business goals or policy’

‘‘We don’t have time / interest / priority’We don’t have time / interest / priority’

The challengeThe challenge

Practical case studyPractical case study

Principal: IT-Solutions Mgr. Solutions ArchitecturePrincipal: IT-Solutions Mgr. Solutions Architecture

Draft Test Strategy guidelines as starting pointDraft Test Strategy guidelines as starting point

My assignment: coaching creation of Test StrategyMy assignment: coaching creation of Test Strategy

Themes: strategy, terminology, organizationThemes: strategy, terminology, organization

Scope: IT Express BeNeLuxScope: IT Express BeNeLux Business, Business-IT and IT-servicesBusiness, Business-IT and IT-services

Roadmap and planningRoadmap and planning

Organize meeting structureOrganize meeting structure

Kick-off with management groupKick-off with management group

Training day for work groupTraining day for work group

Strategy workshops Strategy workshops (4 sessions)(4 sessions)

Elaboration Test Policy & StrategyElaboration Test Policy & Strategy

Reporting to management groupReporting to management group

ApproachApproach

33

55

66

77

11

22

44

Kick-off with managementKick-off with management

Purpose: get feedback and constraintsPurpose: get feedback and constraints

Presentation using provocative argumentsPresentation using provocative arguments

Results:Results: Feedback:Feedback:

How ‘testing’ is perceived in the organizationHow ‘testing’ is perceived in the organization

Effects of insufficient testing (= business case for improvement)Effects of insufficient testing (= business case for improvement)

Important quality characteristicsImportant quality characteristics

ConstraintsConstraints Realistic (acceptable) costs and organization structureRealistic (acceptable) costs and organization structure

33

Strategy workshopsStrategy workshops

1.1. Agree on purpose and scopeAgree on purpose and scope using management feedback and constraints as inputusing management feedback and constraints as input

1.1. Determine policy themesDetermine policy themes reliability, customer facing, usability, reliability, customer facing, usability,

performance, efficient testing, time to marketperformance, efficient testing, time to market

1.1. Definition of testingDefinition of testing

2.2. Business goalsBusiness goals not our responsibility, proactive thinking is (rationale)not our responsibility, proactive thinking is (rationale) 7 values DHL taken as starting point7 values DHL taken as starting point

1.1. Test strategyTest strategy V-model, determine test levelsV-model, determine test levels RACI*, responsibilities, who is ‘main contractor’RACI*, responsibilities, who is ‘main contractor’

55

* = RACI: Responsible, Accountable, Consulted, Informed

Elaboration: resultsElaboration: results

Test Policy documentTest Policy document Quality goals + rationale (2 A4)Quality goals + rationale (2 A4) Test policy (2 A4)Test policy (2 A4)

Test Strategy documentTest Strategy document PrinciplesPrinciples V-modelV-model Characteristics per levelCharacteristics per level Risk based testingRisk based testing

Management presentationManagement presentation Results and recommendationsResults and recommendations

66

Elaboration: Test PolicyElaboration: Test Policy66

Elaboration: V-modelElaboration: V-model66

Full Roll-OUT

Management presentationManagement presentation

RecommendationsRecommendations Agree on common test strategyAgree on common test strategy Apply risk based testingApply risk based testing Install a test competence cellInstall a test competence cell Start a test process improvement projectStart a test process improvement project Agree on common test terminology (e.g. ISTQB)Agree on common test terminology (e.g. ISTQB)

Recommendations have been implementedRecommendations have been implemented Test competence cell: competence manager installedTest competence cell: competence manager installed Interviews / meetings with stakeholdersInterviews / meetings with stakeholders Plan for deployment writtenPlan for deployment written

77

Lessons and success factorsLessons and success factors

It requires time to determine a good test strategy!It requires time to determine a good test strategy!

If there is no policy available, then propose one.If there is no policy available, then propose one.

Involve management (feedback, constraints).Involve management (feedback, constraints).

Broad representation of organization as workgroup.Broad representation of organization as workgroup.

Foundation test knowledge in workgroup (training).Foundation test knowledge in workgroup (training).

Bring sufficient flexibility into your V-model.Bring sufficient flexibility into your V-model.

Who is responsible for what (RACI) ?Who is responsible for what (RACI) ?

Pioneer for implementation from own organization.Pioneer for implementation from own organization.

Epilogue (1)

Follow-up by test competence manager: Further elaboration and deployment of test

strategy (templates and tools)

Organized sessions/training on test awareness

Assistance in projects to apply test strategy

Has set up method to manage test environment

The assignment ended in 2008, what happened afterwards…The assignment ended in 2008, what happened afterwards…

Epilogue (2)

Current situation (mid 2009): Competence cell (0.5 FTE*): central organization

of testing expertise.

Close cooperation helps to get clear requirements

Clearly defined test levels are helpful

Test strategy currently applied in a project for European standardisation of one of DHL’s most important product lines

The assignment ended in 2008, what happened afterwards…The assignment ended in 2008, what happened afterwards…

* = FTE: Full-time equivalent

ConclusionsConclusions

A sound Test Policy and Test Strategy …A sound Test Policy and Test Strategy … are the important starting points for projectsare the important starting points for projects

help to have good discussions …and decisionshelp to have good discussions …and decisions

can be very clear and specificcan be very clear and specific

give answers to relative questionsgive answers to relative questions

are the first step in test process improvementare the first step in test process improvement

… … are worthwhile to give a try in your organization ?!?!are worthwhile to give a try in your organization ?!?!