Stephen K. Allott - People Challenges for Test Teams - SoftTest Ireland

SoftTest Ireland Copyright 2008 ElectroMind Ltd 1

People Challenges for Test Teams

Stephen K. Allott





Introductions

My backgroundAims of this workshop

Explain some common testing challenges based on recent experienceAsk for your experiencesAgree a list of challenges we all face as testersTry to come up with some quick wins & longer term approaches to solutions


My recent clients

Investment bankAirline web siteTravel company web siteMobile phone operator Retail web site Travel company back officeRecruitment web site

Includes assignments done in association with TestHouse (London) and Neueda Technologies (Antrim & Dublin)


Their technologies

Microsoft Dot (.) NET 1/2/3Microsoft VSTS / TFSOracleMS SQL ServerWindows XPCitrix MetaframeQuality Centre / QTP


Number of testers

Typically 12 to 14 in the test teamA couple of clients had only 1, some have none!


Scale 0 1 2 3 4 5 6 7 8 9 10 11 12 13

Key Area Controlled Efficient Optimising

Test strategy A B C D

Life-cycle model A B

Moment of involvement A B C D

Estimating and planning A B

Test specification techniques A B

Static test techniques A B

Metrics A B C D

Test automation A B C

Test environment A B C

Office environment A

Commitment and motivation A B C

Test functions and training A B C

Scope of methodology A B C

Communication A B C

Reporting A B C D

Defect management A B C

Testware management A B C D

Test process management A B C

Evaluation A B

Low-level testing A B C

Composite TPI Test Maturity Matrix from several clients


Summary of TPI results

No documented test processNo formal test policy, strategy or planMany tests, no formal designNo metricsNo proper test environment

Some tested in “live”

No documented test basisNot involved early enough


Some common observations

Constantly changing business prioritiesIncreasing (frantic) demand for new functionalityWeekly, sometimes daily release cyclesAttempt to use automation to solve all the problemsNo numbers to explain testing’s added valueSmall important teams yet with little influence


People challenges – behaviour

MotivationA few are resigned to how it is, no hope?Some expect spoon feedingSome adopt a negative attitudeMany are willing to make improvements, if there was time

Turnover – very high offshoreCulture

Some managers get too involved in detailCo-dependency – testers must push backIt hasn’t broken yet so . . .


People challenges – skills

Testing SkillsProcessesStandardsTechniquesTools

IT SkillsProgrammingDatabasesInfrastructureWeb

Domain KnowledgeIndustry specificApplication knowledgeE-Business

Soft SkillsCommunicationNegotiationManaging technical teams


People challenges – perception

In general testers are not valuedTesters are often blamed

Project delaysPoor qualityManaging expectations with RBT

Lack of awareness of what the test team does


People challenges – managers

Demand for certified staffDemand for skills & knowledgePressure to cut costs and use offshore resourcesDeveloping good test leadsRole of testers in an agile projectSelling the value of testing to the organisation


Themes

Facilitated discussion bearing in mind the following themes:

1. Relentless business pressure for new functionality2. Constant and last minute changes to requirements3. Introduction of new development / testing methodologies e.g. agile 4. Very frequent releases, often weekly, sometimes daily!5. Increasing application complexity over time6. Knowledge gap (e.g. what lies beyond the browser?)7. Steep learning curve (business as well as technical)8. Use of contract and offshore resources (loss of in-house expertise)9. Little or no documentation against which to test10. Lack of attention to all testing requirements (e.g. middleware, back

end database, non-functional quality characteristics)


Discussion – how do we respond?

Business PressuresNew functionalityFrequent changeMaturity

TechnologyComplexRapid changeInterfaces

OffshoreCheaperPlentifulExpertise

BehaviourMotivationAttitudeCulture

SkillsAnalysisIT / TechnicalBusinessTestingSoft skills

ManagersCostsTime to marketAgile


Conclusions

In my opinion testers of the future must . . .Measure / appreciate application complexityBe able to accept and manage last minute changesBe flexible but push back with estimatesIncrease their technical knowledgeLearn more about testing, especially techniquesApply the tried and tested techniquesCapture metrics to help prove the business caseGet involved in the projects much earlier

Recruit / train / certify / develop / coach = RetainShare your views with the group


Final Thoughts / Questions?



[email protected]+44(0) 773 476 1363www.electromind.com



Appendix – Bonus Slides


A typical project in trouble

CR Churn by Month - AveCR=47 AveChurn=150 AveChurnCR=3

1427

58 5565

40

116

279

229213

0 0

254

124

263

0

50

100

150

200

250

300

June July August September October

Months

CR

cou

nt

0

50

100

150

200

250

300

Cha

nges

to C

Rs

CRs Churn Bugs


Bug snapshot by priority

Sev 1 Sev 2 Sev 3 Sev 4 Total

Pri 1 57 81 14 152

Pri 2 19 52 12 1 84

Pri 3 6 14 10 30

Pri 4 2 1 3

None 68 288 114 10 480

Total 146 428 154 21 749


Bug snapshot by resolution

Sev 1 Sev 2 Sev 3 Sev 4 Total

By design 10 37 29 4 80

Duplicate 6 33 7 1 47

External 2 16 4 0 22

Fixed 86 254 95 12 447

Not repro 29 56 4 0 89

Postponed 12 17 6 4 39

Will not fix 1 15 9 0 25

Total 146 428 154 21 749


Impact of the improvements

CR Churn by Month - AveCR=47 AveChurn=150 AveChurnCR=3

1427

58 5565

30

61

17 15 10 5

40

116

279

229213

6686

18 15 10 50 0

254

124

263

13

90

12 9 5 10

50

100

150

200

250

300

June July

Augus

tSep

tembe

r

Octobe

r

Novembe

rDece

mber

Janu

ary

Februa

ry

March

April

Months

CR c

ount

0

50

100

150

200

250

300

Cha

nges

to C

Rs

CRs Churn Bugs

Events Systems1. Vendor2. User Reg.3. User Shop

Sales and Marketing Systems1. Consulting Practice Mgt.2. Vendor Account Mgt.3. Event Mgt.4. Leads and Referral

Databases1. Profile2. Common3. Event OLTP4. Transaction OLTP5. Contact History OLTP6. Domain Reference

World-wide Marketing DB1. Profile OLAP2. Event OLAP3. Analytical Reporting4. Contract History OLAP5. Domain

Corporate Sales and Marketing Systems

System Interfaces1. Event Orders2. Billing & Financials3. Credit Card (real-time)4. Event Communication / FAX5. Direct email service6. Direct mail Postal interface7. Leads & Referrals8. OLTP Staging Interface9. OLAP Staging Interface10. SAP11. Licensing12. Call Center Source: Reality Test Inc.


Basic complexity calculation

Characteristic Value Factor Total

Windows 37 8 296

Objects 23 1 23

Tables 47 1 47

Apps Impacted 3 10 30

Factors are weighted from a testing perspective


Comparison across applications

Application UI Back End

Env Test Code

Middle Tier

Total

Sales Mgr. 1360 2250 152 0 0 3762Call Center 960 824 90 129 0 2003Web Sales 389 183 109 0 2 681 Middleware component

0 146 23 27 232 428


0

1000

2000

3000

4000

PSS CTR

Knowled

ge B

ase

Look

ing G

lass

Web

Sale

s

Accou

nt Man

ager

Delta

Call C

enter

Sales M

anag

er

Application Name

Com

plex

ity V

alue

Complexity comparison


Actual staffing comparison

0

5

10

15

20

PSS CTR

Knowled

ge B

ase

Look

ing G

lass

Web

Sale

sAcc

ount

Manag

er

Delta

Call C

enter

Sales M

anag

er

Applications

Num

ber o

f Tes

ters

Estimated

Actual


Organisation evaluation

Organisation within the companyProcess definition and executionTeamworkAbility to do the job


Organisation within the company

Section Item being evaluated Max Score1.1 Reporting structure 501.2 Professionalism 1701.3 Organisational churn 1351.4a Staffing – permanence 301.4b Staffing – stability 351.4c Staffing – morale 501.4d Staffing – number of testers 401.4e Staffing – organisation type 401.5 Budget control 30


Process definition and execution

Section Item being evaluated Max Score2.1 Software development life-cycle 602.2 Design reviews and documentation 702.3 Early participation in development 502.4 Ability to influence schedule deadlines 602.5 Development standards 602.6 Defined testing approach & procedures 1902.7 Metrics reporting 602.8 Churn 902.9 Patch releases 402.10 Post-project reviews 60


Teamwork

Section Item being evaluated Max Score3.1 Support from production support 603.2 Support from management 603.3 Correct level of quality 903.4 Resource allocation 80


Ability to do the job

Section Item being evaluated Max Score4.1 Adequate equipment / lab 2004.2 Automated testing 704.3 Programming ability 704.4 Non-testing activities 50


Total evaluation score

900

1000

1100

1200

1300

Q1/00 Q2/00 Q3/00 Q4/00 Q1/01 Q2/01 Q3/01

Date

Eval

uatio

n

Total Evaluation Score


Organisational Evaluation

100

200

300

400

500

Q1/00 Q2/00 Q3/00 Q4/00 Q1/01 Q2/01 Q3/01

Date

Eva

luat

ion

Organization

Process

Teaming

Ability


Components of a centre of excellence

Resource pool Test lab

Test Managers SIG

ConsultantsKnowledge Base

R&D Centre

Tool Experts

Project 1

Project 2

Project 3

Project 4


Benefits of a centre of excellence

Independent testingBreadth & depthFewer product recallsFewer production failuresCentral reference pointKnowledgeable staffExpertise across org.Structured career path

Variety of workIdentity, prideMorale, professionalismExtra revenue streamConsistent approachMulti-methodologiesEarlier involvement

Documents

Stephen K. Allott - People Challenges for Test Teams - SoftTest Ireland