TM PM Tutorial

10/14/2014 1:00:00 PM

"End-to-End Testing with the Heuristic

Software Test Model"

Presented by:

Paul Holland

Testing Thoughts

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Paul Holland

Doran Jones, Inc.

Paul Holland is currently the Managing Director of the Testing Practice at Doran Jones. He has

more than nineteen years of hands-on testing and test management experience, primarily at

Alcatel-Lucent where he led a transformation of the testing approach for two product divisions,

making them more efficient and effective. As a test manager and tester, Paul focused on

exploratory testing, test automation, and improving testing techniques. For the past five years,

he has been consulting and delivering training within Alcatel-Lucent and externally to companies

such as Intel, Intuit, Progressive Insurance, HP, RIM, and General Dynamics. Paul is one of

three people who teach the Rapid Software Testing course. For more information visit

testingthoughts.com.

1

End-to-End Testingwith a Heuristic Software Test Model

Paul HollandManaging Director –

Head of Testing Practiceat Doran Jones

My Background

� Independent S/W Testing consultant since Apr 2012

� 19+ years testing telecommunications equipment and reworking test

methodologies at Alcatel-Lucent

� 12+ years as a test manager

� Presenter at CAST, STAREast, STARWest, Let’s Test, STARCanada, and

EuroStar

� Keynote at KWSQA conference & Ho Chi Minh City Software Testing

Conference

� Facilitator of 50+ peer conferences and workshops

� Teacher of S/W testing for the past 6 years

� Teacher of Rapid Software Testing

� Military Helicopter pilot – Canadian Sea Kings 2

2

Attributions

�Many of the concepts that I am presenting come from the Rapid Software Testing class developed by James Bach and Michael Bolton who are often inspired by Jon Bach and Cem Kaner

3

Agenda

�Test Strategy Development – lecture

�Project Statement – incl. Bug Reporting

�Hands-on Test Strategy Development

�Reporting – lecture

�Hands-on Testing and Reporting

�Delivering Final Reports4

3

Test Strategy Development

�Each feature/product is different from other

features/products. Templates != Good.

�Start with “raw” project specific information:

�Detailed description of the elements of the

feature/product

� Identify specific risks related to the project

�Develop a list of “test ideas”

5

Product Coverage Outline

�A Product Coverage Outline can be

described as:

�A parts list

�An inventory of elements (including data)

�Components of a product

�Essentially a list of items of a product that can

be examined in some way

6

4

Benefits of a PCO

� Helps eliminate tunnel vision

�Will help display your thoroughness to others

� Help you gain respect

�Will allow you to get “buy-in” at a high level of what

you are going to consider when testing

� Helps prioritize and divide your testing

� Helps in reporting

7

Product Coverage Outline

How to create a

PCO

8

5

Product Coverage Outline

�Consider the following aspects (SFDIPOT):

� Structure of the program (smallest components)

� Functionality (individual features)

� Data (I/O, create, store, manipulate, backup, …)

� Interface (User interfaces, APIs, …)

� Platform (Computer, CPU, OS, browser, …)

� Operations (How is it used by customers)

� Timings (Race conds, time of day/wk/mth/yr, …)

(Taken from the Rapid Software Testing Course, Bach

and Bolton)

9

Product Coverage Outline

�Consider touring of the feature/product

� Investigate every menu item

� List sub-menus

�Context specific items

� Text boxes and/or Pop-up boxes

� Radio buttons / Check boxes

�Visual elements (pictures, background, logos)

�Complexity (find the most complex thing)

10

6

Product Coverage Outline

� Talk with designers, business people, customers,

customer service, etc.

� Look at previous versions

� Look at marketing information

� Look at competing products

� Read the functional specification document

(*although, I would do this later – seriously)

11

Product Coverage Outline

� As you are investigating the feature/product

� Create a list of ALL the elements that MAY need to be

examined in some way

� Make a mind map (I use XMind)

� Enter the elements into Excel (or a word processor)

� Create a list on paper, flip chart, or whiteboard

� Make a bunch of sticky notes

(I would encourage electronic versions for ease of sharing

and re-use – but it is not mandatory)

12

7

Risks

13

� Identify potential risks to the product that you

might be able to test

� Create a separate list or mind map of risks or add to

the PCO if time is limited

� Identify ways to test the product for the risks

and how to test PCO elements

� Such a combination represents a Test Strategy

� Prioritize test selection then start testing

Our Mission

� Our mission is to discover the quality status of the newly reworked version of linkwok.com.

� We must find critical bugs in Linkwok before this version is deployed. We must advise if there is any good reason to consider postponing the deployment of this release. We are also interested in non-critical bugs, but your preference should be to find critical problems QUICKLY and get them reported SOON.

RECENT CHANGES

� The “sticky note” functionality was rewritten in this release. But, there shouldn't be any problems because the functionality has not changed.

(I will act as the customer representative if you have questions)

14

8

Exercise

�Learn about Linkwok.com

�Create a PCO for Linkwok.com

� This should factor out the components that

MAY be of interest to some testing mission

�You have 30 minutes

�Please email me your PCO: paul@testingthoughts.com

15

Documentation

16

One approach (can be followed or modified or ignored)

� Create a spreadsheet that lists all charters with details

(see next slide)

� Optionally create a Word document as well that lists

the details in a more readable format for customers

� Visual tracking with whiteboard

9

Sample Charter w/Test Ideas

17

ARQ Verification under different RA Modes (approx. 2 sessions)

• Verifying ARQ in all RA modes

• Fixed rate

• Rate adaptive

• SRA -> not supported, should revert to rate adaptive

• Params in XXXX profile should override service profile

Check every parameter in service profile except SRA

Verify no alarms if service profile cannot be met

Does invalid service profile still raise CE alarm if it is overridden

• Does ARQ fixed rate profile follow same rules as old fixed rate service profile?

(i.e. follows planned bitrate)

• Test various rates (very low to very high) in both service profiles

• Will ARQ still work under rate limitations?

• Will ARQ profile with narrow range of min/max rates behave as similar service

profile would?

• Look for holes in successful inits with varying US/DS rates & fall-back rules.

• supported by CO and CPE -> ARQ ok

• supported by CO but not CPE -> falls back to IFEC for both DS + US

• supported by CPE but not CO -> falls back to IFEC for both DS + US

Test Tracking

� An Excel Spreadsheet with:

� List of Charters

� Area

� Estimated Effort

� Expended Effort

� Remaining Effort

� Tester(s)

� Start Date

� Completed Date

� Issues

� Comments

� Does NOT include pass/fail percentage or number of test cases

18

10

Sample Spreadsheet

19

Charter AreaEstimated

Effort

Expende

d Effort

Remaining

Effort Tester Date Started

Date

Completed

Issues

Found Comments

Investigation for high QLN spikes on EVLT

H/W

Performance 0 20 0 acode 12/10/2011 01/14/2012

ALU01617

032

Lots of investigation. Problem was on 2-3

out of 48 ports which just happened to be 2

of the 6 ports I tested.

ARQ Verification under different RA

Modes ARQ 2 2 0ncowan 12/14/2011 12/15/2011

POTS interference ARQ 2 0 0 --- 01/08/2012 01/08/2012

Decided not to test as the H/W team

already tested this functionality and time

was tight.

Expected throughput testing ARQ 5 5 0 acode 01/10/2012 01/14/2012

INP vs. SHINE ARQ 6 6 0ncowan 12/01/2011 12/04/2011

INP vs. REIN ARQ 6 7 5 jbright 01/06/2012 01/10/2012

To translate the files properly, had to install

Python solution from Antwerp. Some

overhead to begin testing (installation,

config test) but was fairly quick to execute

afterwards

INP vs. REIN + SHINE ARQ 12 12

Traffic delay and jitter from RTX ARQ 2 2 0ncowan 12/05/2011 12/05/2011

Attainable Throughput ARQ 1 4 0 jbright 01/05/2012 01/08/2012

Took longer because was not behaving as

expected and I had to make sure I was

testing correctly. My expectations were

wrong based on virtual noise not being

exact.

Weekly Report

�A PowerPoint slide indicating the important

issues (not a count but a list)

� “Show stopping” bugs

�New bugs found since last report

� Important issues with testing (blocking bugs,

equipment issues, people issues, etc.)

� Risks – both new and mitigated

� Tester concerns (if different from above)

� The slide on the next page indicating

progress

20

11

Sample Report

21

0

10

20

30

40

50

60

70

80

90

ARQ SRA Vectoring Regression H/W Performance

Effort

(person h

alf d

ays)

Feature

"Awesome Product" Test Progress as of 02/01/2012

Original Planned

Effort

Expended Effort

Total Expected

Effort

Direction of lines indicates effort trend since last report

Solid centre bar=finished Green: No concernsYellow: Some concernsRed: Major concerns

Final Report

�A complete story that will include:

� Status of the product

� Testing coverage (use PCO as baseline)

� Risk coverage

� “Show stopping” and important bugs

�Other bugs – if warranted to be in report

� Plus tell a compelling story about your

testing…

22

12

23

To test is to compose, edit, narrate, and justifyTHREE stories.

A story about the status of the PRODUCT"

…about how it failed, and how it might fail...

…in ways that matter to your various clients.

A story about HOW YOU TESTED it"

…how you configured, operated and observed it…

…about what you haven’t tested, yet…

…and won’t test, at all, unless our client requests

otherwise…A story about how GOOD that testing was"

…what the risks and costs of testing are…

…what made testing harder or slower…

…how testable (or not) the product is…

…what you need and what

you recommend.

Whiteboard

� Used for planning and tracking of test

execution

� Suitable for use in waterfall or agile (as long as

you have control over your own team’s

process)

� Use colors to track:

� Features, or

� Main Areas, or

� Test styles (performance, robustness, system)

24

13

Whiteboard

� Divide the board into four areas:

� Work to be done (divided in two)

� Work in Progress

� Cancelled or Work not being done

� Completed work

� Red stickies indicate issues (not just bugs)

� Create a sticky note for each half day of work (or mark #

of half days expected on the sticky note)

� Prioritize stickies daily (or at least twice/wk)

� Finish “on-time” with low priority work incomplete

25

Whiteboard Example

26

End of week 1

Out of 7 weeks

14

Whiteboard Example

27

End of week 6

Out of 7 weeks

Exercise

� Develop a quick test strategy to test Linkwok for the

next hour

� I will be asking for brief 30 sec to 1 min interim

3-level test reports

�Create a final 3-level test report (must include PCO& bug list)

�With 30 minutes to go in the class, we will have

maximum 5 minute presentations from each team

28

15

29