M.A.S.Etesting - Let's Testlets-test.com/wp-content/uploads/2013/08/201306... · M.A.S.Etesting (ManualAutomatedScriptedExploratory)! Keith&Stobie& So*ware&Quality&Engineering&Architect,&TiVo&

M.A.S.E testing (Manual Automated Scripted Exploratory)

Keith Stobie So*ware Quality Engineering Architect, TiVo

testmuse.wordpress.com

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

1

Keith Stobie

ASQ CSQE. BBST FoundaHons graduate. ISTQB FL.

Member of AST, ASQ, ACM, and IEEE, also SASQAG and PSQNC

Protocol Quality Assurance Process Model-‐Based Quality Assurance of Protocol DocumentaHon, STVR, 2011

Too Darn Big to Test – ACM Queue vol.3 (#1)-‐Feb.2005 TesHng for ExcepHons, STQE, July/Aug. 2000, 12-‐16

Keynoted at CAST 2007 and MBT-‐UC 2012

Test Architecture, Test Design, Test AutomaHon

Test Methodology

TiVo, Bing, Microso* servers & tools, BEA, Informix, Tandem

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

2

Context

•  Scripted versus exploratory dimension and its interacHon with manual versus automated.

•  Learn about the forces that influence when automaHon or manual tesHng is most appropriate and

•  When confirmatory (scripted) or bug finding (exploratory) is most appropriate.

•  The role and benefit of each type (manual scripted, automated scripted, manual exploratory, automated exploratory).

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

3

Perfect Software Testing

So#ware Tes+ng – Cem Kaner Perfect Tes+ng – James Bach an empirical technical

invesHgaHon conducted to provide stakeholders

with informaHon about the quality

of the product or service under test

TesHng is the infinite process of comparing the invisible

to the ambiguous so as to avoid the unthinkable happening to the anonymous

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

4

tes+ng. IEEE Standard for so*ware and system test documentaHon (IEEE Std 829-‐2008) (A) An acHvity in which a system or component is executed under

specified condiHons, the results are observed or recorded, and an evaluaHon is made of some aspect of the system or component.

(B) To conduct an acHvity as in (A).

Exhaustive vs. Pragmatic Testing Exhaus+ve / Complete Tes+ng

execute program on all possible inputs and compare actual to expected behavior.

•  Could “prove” program correctness.

•  Not pracHcal for any non-‐trivial program.

Comprehensive Tes+ng

•  Usually some (coverage) criteria.

Pragma+c tes+ng

Select a vanishingly small % of all possible tests.

Goal: ExecuHng the Hny % of test will uncover a large % of the defects present.

A tes6ng method is essenHally a way to decide which Hny % to pick.

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

5

Associations

Are you familiar with the term:

Manual Tes+ng?

Automated Tes+ng?

Scripted Tes+ng?

Exploratory Tes+ng?

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

6

It is tradiHonal for testers to be untrained in what I consider the basic cogniHve skills of excellent tesHng. It is tradiHonal, I suppose, for testers to have almost no ability to defend what they do, except through the use of clichés and appeals to authority and folklore. That tradiHon has poorly served our industry, in my opinion. I'd like to replace that with a tradiHon of context-‐driven, skilled tesHng -‐-‐ James Bach hnp://www.saHsfice.com/blog/archives/58

Manual testing What do you think of?

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

7

Automated testing What do you think of?

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

8

Scripted testing

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

9

What do you think of?

A test script in so*ware tesHng is a set of instrucHons that will be performed on the system under test to test that the system funcHons as expected. -‐-‐ hnp://en.wikipedia.org/wiki/Test_script Test script is a set of instrucHons (or steps) to follow in a prescribed sequence.

Exploratory testing What do you think of?

hnp://blogs.msdn.com/b/anunhara/archive/2011/10/20/exploratory-‐tesHng-‐introducHon.aspx

hnp://swtester.blogspot.com/2012/05/what-‐is-‐exploratory-‐tesHng.html

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

10

Agile testing

No test scripts

Adapt as you test Learn as you test Design as you test

Lightweight test Find new bugs all the time

No detailed test planning

Term was coined by Cem Kaner in 1983 in his book TesHng Computer So*ware : “Trust your insHncts” Updated descripHon by James Bach: “Exploratory tesHng is simultaneous learning, test design, and test execuHon, with an emphasis on learning”

Continuum

hnp://swtester.blogspot.com/2012/05/what-‐is-‐exploratory-‐tesHng.html

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

11

Test & Test Case (IEEE)

3.1.39 test:

(A)  A set of one or more test cases.

(B)  A set of one or more test procedures.

(C)  A set of one or more test cases and procedures.

(D)  The acHvity of execuHng (A), (B), and/or (C).

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

12

test case. IEEE Standard for so*ware and system test documentaHon (1) (IEEE Std 610.12-‐1990) A set of test inputs, execuHon condiHons,

and expected results developed for a parHcular objecHve, such as to exercise a parHcular program path or to verify compliance with a specific requirement.

(2) (IEEE Std 829-‐2008) DocumentaHon specifying inputs, predicted results, and a set of execuHon condiHons for a test item.

ScriptedóExploratory Continuum

hnp://www.qualitestgroup.com/media/presentaHons/PPT_Exploratory_TesHng_Explained.pdf

exploratory side of the con6nuum

pure scripted vague scripts

fragmentary test cases charters roles

freestyle exploratory tours

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

13

Itkonen, Juha; Mika V. Mäntylä and Casper Lassenius (2007). "Defect DetecHon Efficiency: Test Case Based vs. Exploratory TesHng" no significant differences in defect detecHon efficiency between {manually execu6ng} test case based tesHng (TCT) and {manual} exploratory tesHng (ET). [However TCT took more 6me – to prepare the test cases] no benefit in terms of defect detecHon efficiency of using predesigned test cases in comparison to an exploratory tesHng approach. – BJ Rollins, hnp://www.sigist.org.il/_Uploads/dbsAnachedFiles/Empirical_EvaluaHoness.pdf

Defect Detection Efficiency: Test Case Based vs. Exploratory Testing 79 advanced so*ware engineering students performed manual funcHonal tesHng on an open-‐source applicaHon with actual and seeded defects.

The distribuHons of detected defects did not differ significantly regarding technical type, detecHon difficulty, or severity.

TCT produced significantly more false defect reports than ET. Surprisingly, our results show no benefit of using predesigned test cases in terms of defect detecHon efficiency, emphasizing the need for further studies of manual tesHng.

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

14

Phase Group 1 Group 2

Prepara+on Test cases for feature set A Test cases for feature set B

session 1 90 min

Test case based tesHng Feature set A

Exploratory tesHng TesHng Feature set A

session 2 90 min

Exploratory tesHng TesHng Feature set B

Test case based tesHng Feature set B

Empirical Evaluation of Exploratory Testing Effectiveness

No significant differences between the two approaches in terms of the detected defect types, severiHes, or detecHon difficulty. Difference in criHcality of defects between the 2 approaches is staHsHcally significant •  ET more likely to idenHfy behavioral issues (e.g., “look and

feel”) •  Scripted tests more likely to idenHfy technical defects

(computaHonal / code level) •  Both approaches may miss criHcal defects The overall effecHveness of both exploratory tesHng and designing effecHve scripted tests depends heavily upon the individual tester’s professional knowledge of the system and tesHng!

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

15

Automated óManual Continuum?

If you read and hand-‐execute the code– if you do exactly what it tells you– then congratulaHons, you will have performed a poor manual test.

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

16

Rule #1B: If you can truly automate a manual test, it couldn’t have been a good manual test.

Rule #1C: If you have a great automated test, it’s not the same as the manual test that you believe you were automa6ng.

Manual Tests Cannot Be Automated

hnp://www.saHsfice.com/blog/archives/58 -‐ James Bach

#1: A good manual test cannot be automated.

“human eyes, ears, hands, and brains are engaged in the execuHon of the test” – Michael Bolton

Technique vs. Approach

•  A technique is a specific method that is used to accomplish a specific goal.

•  An approach is the overall manner in which you act.

•  Exploratory TesHng is an approach to tesHng. •  All test techniques (i.e. manual, funcHonal) can be

done in an exploratory way or a scripted way (predefined test procedures, whether manual or automated)

•  You can work in an exploratory way at any point in tesHng1

hnp://www.qualitestgroup.com/media/presentaHons/PPT_Exploratory_TesHng_Explained.pdf

1 hnp://www.tesHngeducaHon.org/BBST/exploratory/BBSTExploring.pdf

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

17

Random vs. Ad Hoc vs. Exploratory

hnp://qatestlab.com/services/No-‐DocumentaHon/ad-‐hoc-‐tesHng/

“To the extent that the next test we do is influenced by the result of the last test we did, we are doing exploratory tesHng. “ – James Bach

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

18

Coder -‐ Domain Expert continuum

Manual

Coder Domain Expert

scriptless flowchart

Keyword AcHon word Gherkin

Rspec FIT

xUnit

test libraries

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

19

Automated

Automation goal

hnp://www.methodsandtools.com/archive/archive.php?id=94

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

20

Exploratory – find bugs does not guarantee any type of coverage

Numerous heurisHcs

Scripted – ConfirmaHon requires some knowledge of upfront results

Automated – requires hardware and so*ware may miss side-‐effect issues

Manual – requires human (even if computer assisted) may miss minor issues .

Continuum combinations

Automated Exploratory Manual Exploratory

Automated Scripted Manual Scripted

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

21

Keith’s History & Experience Mostly an automated script tester of Systems and Services (backend).

Wrote tools for exploratory API tesHng.

Did large scale system tesHng using random load generators.

Rare bouts of manual tesHng: Bing, Doyenz, TiVo

Three years of Model Based TesHng to automaHcally generate test scripts.

Limited GUI tesHng (manual or automated) [not my focus]

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

22 Manual Scripted: 5% Automated Scripted: 85%

Manual Exploratory: 5% Automated Exploratory: 5%

Manual Exploratory

Bing – Defects every week

Ken Johnston Single lener queries – inappropriate results

Bing log analysis : Big Data

Abandonment analysis – navigaHonal queries on images.

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

23

Google “Z” safe off

Bing “s” (moderate)

Search Images Navigation

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

24

TiVo Manual Exploratory bugs

Take home builds

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie,

TiVo

25 Click

Charter Login

Outline Enter Username and Password

Specific

•  Click mouse in username field to place cursor in username field. Type “testuser”.

•  Click mouse in password field to place cursor in password field. Type “Pa$$w0rd”

•  Click on login bunon

20 Ju

ne 2013

26

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

Manual Detail

20 Ju

ne 2013

27

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

AcHon LoginEnteringUsernameAndPassword( string username, string password)

Specific

• Mouse.Position(username).Click Username.Enter(“testuser”)

• Mouse.Position(password).Click Username.Enter(“Pa$$w0rd”)

• LoginButton.Click

Automated Details

TiVo Manual Script

Objec+ve

Verify that the TCD is able can make a recording from live TV Steps 1.  Press the Live TV bunon 2.  Press the Channel Up bunon to change channels and verify A 3.  Wait awhile to build up some video in the cache 4.  Press the Record bunon 5.  Press select on the "Record this showing" menu item 6.  Allow some Hme to pass 7.  Press the Record bunon 8.  Press the down bunon then select to select "Stop the current recording" 9.  Press the TiVo bunon (you'll go to the main screen) 10.  Press the Select bunon. You should be on My Shows (or what ever its name for that

product) 11.  Verify B

Expected Results A. Verify the new channel has video that is correctly displayed B. Verify the recorded show is present at the top of the show list if you are sorted by date

Test Case 120527: Record from Live TV

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

28

TiVo Manual Script

Objec+ve

Verify able to direct tune cable channels.

Steps

1. key in a cable channel number from liveTV such as 201 2. verify A

Expected Results

A. Able to tune cable channel

Test Case 108864 Direct Tuning -‐ cable channels

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

29

TiVo Automated script

$controller-‐>gotoScreen('central’)

$controller-‐>gotoLiveTVChannel($channel);

$result = $controller-‐>{'screendump'} -‐>parseScreenDump(filefilter => "livetv.xsl");

$tuned_channel = $result-‐>{'channel-‐number'};

if ( $channel ne $tuned_channel ) {

$self-‐>setError("Not able to tune to Channel.");

return FAILURE;

}

return SUCCESS;

Test Case 108864 Direct Tuning -‐ cable channels

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

30

Automated Scripted Testing Pros •  Low learning curve, no special tools •  Low risk, done it for years •  Reach, everyone can do it, no tools required •  Predictable, repeatable •  Success, proven mechanism to find bugs

Cons •  Tests require a lot of code, every scenario •  Expensive to maintain, cleanup actually lasts for years •  Test issues, far surpass product bugs •  Low coverage, out of the box •  Rigid, difficult to improve, change, or just keep up to date •  StaHc, stop finding bugs, same scenarios, missing bugs •  Doesn’t scale, increasing feature sets, complexity, dependencies

Ed Triou

20 J

une

201

3

31

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

Copyright © 2013 Keith Stobie, TiVo 32

What’s a Model?

A model:

Is an abstrac+on or simplified representa+on of the system from a parHcular perspecHve

Supports inves+ga+on, discovery, explanaHon, predicHon, or construcHon

May be expressed as a descripHon, table, graphical diagram, or quanHtaHve mathemaHcal model

Is not necessarily comprehensive

20 Ju

ne 2013

Copyright (c) 2002 Elisabeth Hendrickson, Quality Tree So*ware, Inc.

33

Models in Everyday Life

Inspired by :“Modeling: A Picture's Worth 1000 Words” Copyright (c) 2002, Quality Tree So*ware, Inc.

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

// Some WCF service interface [ServiceContract] public interface IService { [OperationContract(AsyncPattern = true)] IAsyncResult BeginGetCustomers(AsyncCallback callback , object state); List<Customer> EndGetCustomers(IAsyncResult result); }

Manual Model Exploration

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

34

Simple Finite State Machine (FSM) of an API.

End API

beginAPI

Not Started

endAPI

begin API

Proces-‐sing

Test Cases (input sequences) 1.  begin; end; end 2.  begin; begin 3.  end; 4.  begin; end; begin; end

hnp://www.codeproject.com/ArHcles/56175/A-‐Generic-‐Class-‐for-‐Wrapping-‐Asynchronous-‐Begin-‐En

Automated Exploratory

Accidental – thronled process creaHon

Dumb Monkey & Fuzzing – Random input

Random failures – similar to Chaos monkey

Random stress tesHng

Mine

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

35

Fuzz Testing

History – a dark & stormy night

Fuzz Revisited: A Re-‐examina6on of the Reliability of UNIX ... by BP Miller, et al

hnp://www.opensource.org/advocacy/fuzz-‐revisited.pdf

hnps://www.owasp.org/index.php/Fuzzing Tools list

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

36

providing invalid, unexpected, or random data to the inputs of a computer program

Fuzz Testing

Goal: push invalid data as far into system as possible. Unexpected input : length and content Both Dumb and Smart fuzzing valuable! •  Dumb generates data with no regard to the format (data form of dumb Monkey tesHng)

•  Smart requires knowledge of data format or how data is consumed. The more detailed the knowledge – the bener.

Generate -‐ creates new data from scratch

Muta+on -‐ transforms sample input data to create new input data : add/change/delete data Most fuzzing tools are a mix of each approach

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

37

Monkey Testing

Dumb / Wild / Unmanaged

banging on the keyboard by randomly generaHng input (key-‐presses; and mouse moves, clicks, drags, and drops)

Smart / Trained / Managed

detects available opHons displayed to the user and randomly enters data and presses bunons that apply to the detected state of the applicaHon

Chaos Monkey (ne�lix.com) hnp://techblog.ne�lix.com/2012/07/chaos-‐monkey-‐released-‐into-‐wild.html

Chaos Monkey is a service which runs in the Amazon Web Services (AWS) that seeks out Auto Scaling Groups (ASGs) and terminates instances (virtual machines) per group.

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

38

hgp://www.ques+oningso#ware.com/2007/02/people-‐monkeys-‐and-‐models.html

Flavors of Stress Tests

Load Ensures that system can perform “acceptably” under Input and Resource load both Constant and Varying

Limit When capacity\limits are known. Equivalent to boundary testing

Breakpoint When capacity\limits are not known. Find heaviest load the component can handle, before failure

Torture Taking component beyond known or pre-defined min\max limits

Duration To run for some amount of time without undue load or failures. (aka “soak” testing)

Synchronization Flush out timing\synchronization\concurrency issues, e.g., multi-thread tests

Fault Injection Deterministically inject faults, simulate error conditions

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

39

40

Chat Slice where local consistency does matter

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

Spec Explorer

Move to a parHcular state (point), and then explore it (shoot).

Also “On The Fly” tesHng – conHnue generaHng inputs unHl Hme runs out. (Smart “state” monkey).

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

41

point and shoot hgp://msdn.microso#.com/en-‐us/library/ee620432.aspx

Manual Testing Advantages Disadvantages

Leverage experience of Testers to find Defects

Hme consuming

usable for majority of so*ware tesHng types

not suitable for Performance TesHng, Stress TesHng and Soak TesHng

performable in different phases of SDLC

ROI Is low for tests like Smoke or Regression Tests

performable even when applicaHon is not stable

Can lead to defect leakage due to human miss or error

EffecHve when requirements are volaHle

Not effecHve if tests require validaHon of large data sets

EffecHve when UI changes are frequent

Cannot perform most of the tests related to Security TesHng

Testers can lose interest running same tests over longer period of Hme

Success depends on experience and knowledge of testers

www.So*wareTesHngSo*ware.com

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

42

Manual/Automated Effectiveness

Manual Effec+ve Automated Effec+ve FuncHonal Sanity Unit Smoke Exploratory Database Ad-‐Hoc Regression IntegraHon API User Acceptance Web Services Usability Alpha / Beta Compliance

hnp://www.so*waretesHngso*ware.com/manual-‐tesHng/

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

43

Exploratory Tes+ng

•  FuncHonality 1

•  FuncHonality 2 •  FuncHonality 3

•  FuncHonality 4 Scripted Tes+ng





Differences Between Scripted and Exploratory Testing Scripted Tes+ng Exploratory Tes+ng

Directed from requirements Directed from requirements and exploring during

DeterminaHon of test cases well in advance

DeterminaHon of test cases during tesHng

ConfirmaHon of tesHng with the requirements

InvesHgaHon of system or applicaHon

Emphasizes on predicHon and decision making

Emphasizes on adaptability and learning

Involves confirmed tesHng Involves InvesHgaHon

Is about Controlling tests Is about Improvement of test design

Like making a speech — you read from a dra*

Like making a conversion — it’s spontaneous

The script is in control The tester’s mind is in control

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

44

hnp://tesHnginterviewsquesHons.blogspot.com/2013/01/exploratory-‐tesHng.html

BeneUits of Computer-‐Assisted Testing •  Easy test case maintenance

•  Reduced costs •  More test cases

•  Early bug detecHon •  Increased bug count •  Time savings

•  Time to address bigger test issues

•  Improved tester job saHsfacHon

Harry Robinson Exploratory Test AutomaHon slides from CAST 2010

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

45

“You can start with a small model and build in the direcHon you think will do the most good for you. “

Issues with Random Testing

VerificaHon of the result. Easier to have staHc tests with pre-‐computed outputs.

Very low probability sub-‐domains

•  likely to be disregarded by random tesHng but cost of failures in those sub-‐domains may be very high

•  good reason for using it as a complementary rather than the sole tes+ng strategy.

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

46

Validation & VeriUication

ValidaHon (requirements confirmaHon) • Establishing the fitness of a so*ware product for its use. • “Are we building the right product?” • Requires interacHon with customers

VerificaHon (design confirmaHon) • Establishing the correspondence between the so*ware and its specificaHon.

• “Are we building the product right?” • Requires interacHon with so*ware

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

47 hnp://www.construx.com/PracHcing_Earl/Doing_JusHce_to_V_V/ … Confirmaiton

Analyze these claims:

You should write a test plan

It’s important that tesHng be repeatable

Each test case should have an expected result

Test automaHon saves money and Hme

All tesHng is based on a model of what is being tested

Good enough quality is not good enough

An undocumented test cannot be improved

Exploratory tesHng is a useful pracHce

It’s bener to use the term defect than bug

Ambiguity should be removed from requirements

Becoming a Test Expert – James Bach

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

48

ConUirmation & Bug Finding

Confirm Find bugs Quality isn’t worse than before Quality isn’t sufficient Support the team CriHque the product Repeatable Unpredictable

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

49

FuncHonal Tests Examples Story Tests Prototypes SimulaHons

Exploratory TesHng Scenarios

Usability TesHng User Acceptance TesHng

(UAT) Alpha / Beta

Unit Tests Component Tests

Performance & Load TesHng

Security TesHng “ility” TesHng

Supp

ortin

g th

e Te

am

Critique Product

Business Facing

Technology Facing

Q2 Q3

Q1 Q4

Copyrig

ht 2009: Lisa

Crispin, Ja

net G

rego

ry – Drago

nFire

Inc

used with

permission.

Agile Testing Quadrants

hnp://waHrmelon.com/2011/06/10/yet-‐another-‐so*ware-‐tesHng-‐pyramid/

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

50

Automated & Manual Manual

Automated Tools

51

Construx Validation Test list Level Test Type Description Notes

0 Unit Do individual code units (i.e., modules) work by themselves?

Typically done by developer to check the integrity and functionality of code units.

0 Code Integration

Do individual code units work well with one another?

Typically done by developer to insure code units interface successfully with one another.

1 Build Acceptance

Can the application be successfully installed?

Done by Tester

1 Smoke Does the build work well enough to be tested?

Done by Tester

1 Regression Have bug fixes or design enhancements in the code caused other parts of the program to malfunction?

Executed immediately after a successful Smoke Test. Tester reruns tests proved successful in the previous build.

2 Functional Are all the features and functions described in the Functional Specifications implemented and working properly?

Should cover both Front End (UI, GUI) as well as Backend (i.e., data validation). Conducted by QA as Black Box testing.

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

52


3 Desktop Configuration/Host Integration

Does the program: 1) Work in the communications, hardware and software environment described in the Requirements documents? 2) Run successfully with other typical programs loaded? 3)  Interface successfully with remote network resources (e.g., database servers) and local peripherals (e.g., printers and modems)?

Integrates the program into the hardware environment, confirms that it can run concurrently with other typical software and that all necessary resources (local and remote) can be accessed. Conducted by QA as Black Box testing.

4 Load / Performance

Does the program: 1) Work acceptably fast for the user under typical conditions? 2) Work acceptably well when subject to extreme processor activity or volumes of inputs/outputs? 3) Work acceptably well with limited resources?

Finds out how well the program handles under less than optimal conditions. Managed by QA but often involves automation or volunteer ad hoc testers and network personnel to create atypical conditions.

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

Cop

yrig

ht ©

201

3

Kei

th S

tobi

e, T

iVo

53


4 Security / Compliance

Has end to end system security been confirmed? Does the program threaten to expose: 1)  Corporate assets? 2)  Private Customer Information? Does the product threaten government and organizational policy/procedure.

Insures protection of corporate assets and private user information from unauthorized access and assures compliance with governmental, industry and organizational policies and procedures.

5 Accept-ance

Is the service: 1)  Comfortable to use? 2)  Customizable? (if applicable) 3)  Complete? Correct? (includes all user desired features and functions) 4)  Control flow driven: efficiently handles typical user scenarios 5)  Error flow driven: offering options and instructions for every error path.

Checks the program for ultimate user satisfaction.

6 Test in Production

1)  Does the service operate correctly with real load

Monitors the system for continual satisfaction

20 Ju

ne 2013

Static / Dynamic V&V

Sta+c

•  So*ware inspecHons •  StaHc analysis of source code

•  control/data flow analysis

Dynamic

•  Defect tesHng – look for errors in funcHonality •  Load tesHng – look for errors in scalability, performance,

reliability.

Verifica+on(design) & Valida+on(requirements)

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

54

Software Testing Techniques

StaHc

So*ware tesHng techniques

Dynamic

SyntacHc SemanHc Black-‐box White-‐box

Structural Random FuncHonal

StaHsHcal DeterminisHc

must execute the so*ware?

examine code syntax?

code examine?

How is test data selected?

Test data type generated?

Discrete ConHnuous Fault-‐based Error-‐based

Exit Criterion (reliability model)

Adequacy Criterion (control / data flow, …)

An EvaluaHon Scheme of So*ware TesHng Techniques, H-‐D. Chu, Univ. of Newcastle, Dept. of CS, TR 583, June 1997

Classifica3on Evalua3on

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

55

no

no no yes

yes

yes

What do you think?

•  Is it “tesHng” if you simply run the program to see what it does?

•  Is requirements confirmaHon done best with staHc or dynamic tesHng?

•  Is design confirmaHon done best with staHc or dynamic tesHng?

A Beginners Guide to TesHng – Phillip Johnson, InformaHon & CS, U. of Hawaii

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

56

… Testing Acceptance Accessibility AcHve Agile Age Ad-‐hoc Alpha AsserHon API All-‐pairs Automated Basis Path Backward CompaHbility Beta Benchmark Big Bang IntegraHon Binary Portability Black box Boundary Value Bonom Up IntegraHon Branch

Breadth Business Logic Code-‐driven CompaHbility Comparison Component ConfiguraHon CondiHon Coverage Compliance Concurrency Conformance Context Driven Conversion Decision Coverage DestrucHve Dependency Dynamic Domain Error-‐Handling End-‐to-‐end Endurance Exploratory

Equivalence ParHHoning Fault injecHon Formal verificaHon FuncHonal Fuzz Gorilla Gray Box Glass box GUI so*ware GlobalizaHon Hybrid IntegraHon IntegraHon Interface Install/uninstall InternaHonalizaHon Inter-‐Systems Keyword-‐driven Load LocalizaHon Loop Manual Scripted Manual-‐Support

Model-‐Based MutaHon Modularity-‐driven Non-‐funcHonal NegaHve OperaHonal Orthogonal array Pair Passive Parallel Path PenetraHon Performance QualificaHon Ramp Regression Recovery Requirements Security Sanity Scenario Scalability Statement

StaHc Stability Smoke Storage Stress Structural System System IntegraHon TesHng in ProducHon Top Down IntegraHon Thread Upgrade Unit User Interface Usability Volume Vulnerability White box Workflow

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

57

Manual Exploratory

+ Cost effecHve one-‐shot bug discovery

-‐  No coverage guarantees, varies every Hme (for bener or worse)

Tester requires extensive background or knowledge

Numerous heurisHcs

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

58

Manual Scripted

for rapidly changing product + can guarantee some coverage

⬇ Manual exploratory doesn’t guarantee coverage

⬇ Automated scripted maintenance can be too high

+ Costly automaHon vs. cheap human Moravec's paradox : reasoning vs. sensorimotor,

e.g., sound quality, video quality, captcha -‐ May miss “minor” changes, e.g., typos, wrong font, …

•  Requires upfront result knowledge Asperger syndrome highly talented specialist people

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

59

Automated Scripted

+ Guaranteed coverage for slowly or non-‐changing product

-‐  Misses many “side” observaHons, and/or very fragile

•  Requires upfront result knowledge

⬇ Manual Scripted expensive vs. repeated automaHon

•  Manual Exploratory complements large uncovered areas

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

60

Regression vs Latent

WriHng tests and running tests both have costs.

Run same tests, reduce regression defects. Regression: old code fails due to new code

AutomaHon mostly helps here

Create new tests, reduce latent defects (always there). Manual tesHng can most help here

Regression Latent

Old Code New Code

20 Ju

ne 2013

61

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

Regression Risk vs Code Base

0%

50%

100%

0

50

100

150

200

250

1 2 3 4 5 6 7 8 9 10 11

Old Code New Code Regression Risk

Example for 10 iteraHons when each new 20 units of code has 5% errors and old code 1% regression errors.

20 Ju

ne 2013

62

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

IteraHon / Sprint / Release

% of bugs due to regression

Units of Code


+ Finds defects that other approaches miss

-‐  Incomplete, can’t cover most of products today

•  Numerous heurisHcs

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

63

For each Type or Technique an approach to testing

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

64


Pex (pexforfun.com) ModelJUnit, geneHc Modifying in the debugger

Many tools, e.g. xUnit Debugger script

Automated Scripted Manual Scripted Unit / Structural


Broad, not deep Primary paths

(for slow changing) most basic use cases (for fast changing)


Smoke


Model Based TesHng heurisHcs

for most common/criHcal no ROI to automate


Func+onal / Black Box

Automated Unit Testing Current automaHc unit generaHon test techniques: •  Simple known failure modes (e.g., null parameters as inputs)

•  analysis of the data structures used. Data structure generaHon can be based on Constraint Solvers, Models, and symbolic execuHon

You may see 100% code coverage, with no results checking! Be aware of results checking thoroughness; not just code coverage.

Eclat creates approximate test oracle from a test suite, then uses it to aid in generaHng test inputs likely to reveal bugs or expose new behavior while ignoring those that are invalid inputs or that exercise already-‐tested funcHonality

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

65

Manual Script Exactness

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

66

Avoiding Overkill in Manual Regression Tes+ng -‐ Lisa Shepard hnp://www.uploads.pnsqc.org/2012/papers/t-‐46_Shepard_paper.pdf

Document the Intention of the Test

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

67

Avoiding Overkill in Manual Regression Tes+ng -‐ Lisa Shepard

Jonathan Kohl, hnp://www.methodsandtools.com/archive/archive.php?id=65

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

68

Manual Exploratory

Marick hypothesis:

An applicaHon built with programmer TDD, whiteboard-‐style and example-‐heavy business-‐facing design, exploratory tes+ng of its visible workings, and some small set of automated whole-‐system sanity tests will be cheaper to develop and no worse in quality than one that differs in having minimal exploratory tes+ng, done through the GUI, plus a full set of business-‐facing TDD tests derived from the example-‐heavy design.

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

69

hgp://www.exampler.com/blog/2008/03/23/an-‐alterna+ve-‐to-‐business-‐facing-‐tdd/

Automated Exploratory Testing

GeneHc Algorithms -‐-‐ global opHmizaHon vs random

Unit, FuncHonal, Concurrency

Guided -‐-‐

Kyle A. Larsen – coverage exploraHon of virus detecHon

Don Sleuts – SQL grammar & negaHve grammar

Large (e.g. nested) SQL legal inputs

SQL almost correct (SQL monkey, Fuzzing)

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

70

Basic outline of Genetic Algorithm

IniHalize (populaHon) -‐-‐ global opHmizaHon

Evaluate (populaHon)

While (stopping condiHon not saHsfied) { SelecHon (populaHon) -‐-‐ most fit => most likely to swap Crossover (populaHon) -‐-‐ swap “genes” Mutate (populaHon) -‐-‐ small changes to small group Evaluate (populaHon) -‐-‐ kill off less fit

}

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

71

Input vars == genes, set of inputs == chromosomes.

A Survey on So*ware TesHng Techniques using GeneHc Algorithm JCSI InternaHonal Journal of Computer Science Issues, Vol. 10, Issue 1, No 1, January 2013 ISSN (Print): 1694-‐0784 | ISSN (Online): 1694-‐0814 hnp://ijcsi.org/papers/IJCSI-‐10-‐1-‐1-‐381-‐393.pdf

Testing real-‐time systems using genetic algorithms use geneHc algorithms to find inputs with the longest or shortest execuHon Hmes automaHcally.

check whether they produce a temporal error.

The fitness func+on is the execuHon Hme measured in processor cycles.

Experiments using geneHc algorithms on a number of programs with up to 1511 LOC and 843 integer input parameters have successfully idenHfied new longer and shorter paths than had been found using random tesHng or systemaHc tesHng.

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

72 So*ware Quality Journal, 1997, Volume 6, Issue 2, pp 127-‐135

Machine vs. Human

Machine Human Chess Play (deep search) Chess Play (panern match) Learning (training data) Learning Exploring Exploring

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

73

Advanced Chess is a form of chess developed in 1998 by Kasparov where a human plays against another human, and both have access to computers to enhance their strength. Argued by Kasparov to be stronger than a human or computer alone

What Is Exploratory Test Automation? “Computer assisted tesHng

that supports learning of new informaHon

about the quality of the so*ware under test”

-‐-‐ Douglas Hoffman & Cem Kaner

“automated” tests are enHrely scripted tests – Michael Bolton

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

74

For each Type or Technique an approach to testing

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

75


(aka Stress!) Poke & Measure

Many tools “exercises”


Performance


Reachability, GUITAR, AUTO Works for me?

Automated checks Specifics to look for


Usability


Dynamic fuzzing Social anacks

Script kiddies Captcha


Security

Memes about Test Automation

“Automated tests represent the automaHon of a manual process.” – Harold F. Tipton, Micki Krause

“The most important benefit of automated tesHng over convenHonal manual tesHng is the minimizaHon of costs over repeated tests.” – Markus Helfen, Michael Lauer

“I have never been convinced that finding ‘new’ bugs is a realisHc expectaHon for test automaHon.” – I.M. Testy

“A*er you have run your automated tests for the first Hme, you are done finding new bugs. Never again will you find a new issue. ” – Steve Rowe

“Running automated test scripts can not be used to find new bugs in the so*ware ...” – Cordell Vail

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

76

The Net=lix Simian Army

Chaos Monkey randomly disables producHon instances Freely available on github

Latency Monkey induces arHficial delays

Security Monkey finds security violaHons or vulnerabiliHes

10-‐18 Monkey (short for LocalizaHon-‐InternaHonalizaHon, or L10n-‐i18n) detects configuraHon and run Hme problems in instances serving customers in mulHple geographic regions, using different languages and character sets.

Chaos Gorilla simulates an outage of an enHre Amazon availability zone.

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

77

hgp://techblog.nenlix.com/2011/07/nenlix-‐simian-‐army.html

TestIstanbul Conferences 2012

Hybrid Approach

•  Hybrid approach combines pure random tesHng with heurisHc search strategies in interleaving phases to increase overall coverage

•  User can choose between various coverage goals, exit criteria and search strategies

Random Search Phases

Heuris+c Search Phases

An InnovaHve Approach to Model-‐based TesHng

Fake TrafUic (capacity test)

User Requests Service Backend

service Frontend service

Data Data Feed

ProducHon

Test

replayed Requests

Capacity test a single service, or enHre offering (via the frontend)

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

79

20 Ju

ne 2013

Modern Cap Generation Toolset

Controller

DC1 Monitoring system

Capacity clients

Bing Entry point VIP


Capacity clients

Bing entry point

VIP


Capacity clients

Bing Entry point VIP

DC…n Monitoring system

Capacity clients

Bing entry point VIP

From: Greg Veith – Workshop on Performance & Reliability 2010 (WOPR14)

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

80 1.  Controller drives each of the clients across any/all DCs 2.  Capacity is generated within each datacenter to avoid cross-‐DC network uHlizaHon 3.  Controller pulls data from monitoring system auto-‐throgling based on SLA 4.  2 -‐3 engineers required for a typical full scale test

20 Ju

ne 2013

Harry Robinson

Consistency HeurisHc

Birthdates

Abraham Lincoln 2/12/1809

Abe Lincoln 12/2/1809

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

81

Rules of Thumb

Exploratory Test Automation Examples 1.  Disk buffer size 2.  Simulate events with diagnosHc probes 3.  Database record locking 4.  Long sequence regression tesHng 5.  FuncHon equivalence tesHng (sample or exhausHve comparison to a

reference funcHon) 6.  FuncHonal tesHng in the presence of background load 7.  HosHle data stream tesHng 8.  Simulate the hardware system under test (compare to actual system) 9.  Comparison to self-‐verifying data 10.  Comparison to a computaHonal or logical model or some other oracle 11.  State-‐transiHon tesHng without a state model (dumb monkeys) 12.  State-‐transiHon tesHng using a state model (terminate on failure rather

than a*er achieving some coverage criterion) 13.  Random inputs to protocol checkers

www.kaner.com/pdfs/highvolCSTER.pdf hnp://www.associaHonforso*waretesHng.org/wp-‐content/files/Kaner_Hoffman-‐ExploratoryTestAutomaHon.pdf

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

82

For More Information

Keith Stobie

TiVo

hnp://testmuse.wordpress.com/

[email protected]

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

83

References

hnp://blogs.msdn.com/b/anunhara/archive/2011/10/20/exploratory-‐tesHng-‐introducHon.aspx hnp://tesHnginterviewsquesHons.blogspot.com/2013/01/exploratory-‐tesHng.html hnp://waHrmelon.com/2011/06/10/yet-‐another-‐so*ware-‐tesHng-‐pyramid/ hnp://swtester.blogspot.com/2012/05/what-‐is-‐exploratory-‐tesHng.html hnp://www.saHsfice.com/arHcles/what_is_et.shtml hnp://www.goforexam.in/2012/01/what-‐is-‐difference-‐between-‐scripted.html hnp://www.methodsandtools.com/archive/archive.php?id=65 hnp://www.tesHngeducaHon.org/BBST/exploratory/BBSTExploring.pdf

Exploratory Tes+ng

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

84

References

hnp://www.associaHonforso*waretesHng.org/wp-‐content/files/Kaner_Hoffman-‐ExploratoryTestAutomaHon.pdf

kaner.com/pdfs/VISTACONexploratoryTestAutma6on.pdf

www.kaner.com/pdfs/highvolCSTER.pdf

Harry Robinson Exploratory Test AutomaHon slides from CAST 2010


20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

85

References

Hendrickson, E. “A Picture’s Worth 1000 Words,” STQE Magazine, September/October 2002

PNSQC 2002

Winant, B. “Modeling PracHce and Requirements,” S6ckyminds.com. Available: hnp://www.sHckyminds.com/se/s3565.asp Harry’s Model-‐Based TesHng website:

hnp://www.geociHes.com/model_based_tesHng/

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

86

Model Based Tes+ng

References

hnp://www.so*waretesHngso*ware.com/manual-‐tesHng/

hnp://www.methodsandtools.com/archive/archive.php?id=94

hnp://qatestlab.com/services/We-‐Are-‐Professionals-‐in/automated-‐tesHng/

Manual vs Automated

20 Ju

ne 2013

Copyrig

ht ©

2013 Ke

ith Stobie, TiVo

87

Documents

M.A.S.Etesting - Let's Testlets-test.com/wp-content/uploads/2013/08/201306... · M.A.S.Etesting (ManualAutomatedScriptedExploratory)! Keith&Stobie& So*ware&Quality&Engineering&Architect,&TiVo&