Real life test; real life situations

|

Real-life test; Real-life situations

A practical example of the Statistical Usage TestDutch Testing Conference 2014André Verschelling; Sr. test Consultant High Tech

1Real-life test; Real-life situations

|

Customer satisfaction

Number one need: Involvement Customers want to feel like

you care about their needs,and that you’re emotionallyinvested in helping themsolve their problems(Source: http://www.businesspundit.com/top-10-things-that-all-customers-want/)


http://www.businesspundit.com/top-10-things-that-all-customers-want/

http://www.businesspundit.com/top-10-things-that-all-customers-want/

|

Therefore, we test

Because testing … … is a process that provides insight into, and

advice on, quality and the related risks

And quality …… is the totality of features and characteristics

of a product or service that bear on its ability to satisfy stated or implied needs

Thus, …… the customer gets what he really needs, right?


|

What is happening


Focus shift

|

Does that satisfy the customer?

“On system level we do exploratory testing”

Does that really address the customer needs …

… and does it solves his/her problems?


|

What about test design techniques?


Test Design Technique

Useful for Test Design Technique

Useful for

Decision Table Verification Evolutionary Algorithms

Verification

Classification Tree Method

Verification State Transition Test

Verification

Elementary Comparison Test

Verification Real-life test: Statistical usage Test

Verification & Validation

Error Guessing Verification Rare Event Test Verification

Exploratory Testing

Verification & Validation

Semantic Test Verification

Data Cycle Test Verification Syntactic Test Verification

Process Cycle test Verification Use Case Test Verification & Validation

|

Exploratory Testing Only explores what the testers has interest in Tester in center stage, instead of user Actually verifying, not validating

Use Case Testing Fixed and pre-defined workflows to represent

user behavior Based upon defined actors/users

Real-life test: Statistical Usage Test Statistically correct user behavior randomly

applied Based upon operational usage of all users


Validation Techniques

|

Statistical Usage Test

A truly real-life test Operational Usage is captured in Operational

Profiles Which operation is most likely to be performed

under which circumstances?• Events • ‘States’ (history classes)• Probability of state/event combination


Event 1 Event 2 … Event m

History Class 1

P1,1 P1,2 .. P1,m

History Class 2

P2,1 P2,2 .. P2,m

… .. .. .. ..

History Class n

Pn,1 Pn,2 .. Pn,m

| 9Real-life test; Real-life situations


Example of an operational profile (PVR/VCR)

Events ►History classes ▼

evStop evRewind(X tics)

evPlay(X tics)

evFastForward(X tics)

evRecord(X tics)

Standby 0 0.2 0.5 0.2 0.1

Rewind 0.3 0 0.6 0.1 0

Play(counter < 15000)

0.5 0.3 0 0.2 0

Fast forward 0.5 0.1 0.4 0 0

Record 1 0 0 0 0

Play(counter ≥ 15000)

0.8 0.15 0 0.05 0


Gather the Operational Profile (1)

Formal analysis / specification Differentiate customers Differentiate users Differentiate system modes Differentiate functions Specify operational profile

for each [customer, user, system mode, function]

Learn by example Watch and record Interviews


Gather the Operational Profile (2)

Better: Use production data Log files Transaction data Usage data

Most systems are evolutions instead of revolutions Evaluate predecessors Evaluate comparable systems



Test case starts with initial state (i.e. History Class

1) Randomly choose next event Example Test Case for the given Operational

Profile(= sequence of transitions):

Test case must continue until execution profile corresponds with Operational Profile

Standby(current count = 15000) | evRewind(10000 Tics) | Rewind(5000) | evPlay(6000) | Play(11000) | evStop | Standby(11000) | evPlay(5000) | Play(16000) | evRewind (8000) | Rewind(8000) | evPlay(3000) | Play(11000) | etc., etc., etc.

|

Everything your users do is done prove it

No fixed sequence, butrandom selection at runtime statistically correct

Thus, caring about the users’needs

But what if the real user finds an anomaly?


Run the test

|

OK, there’s more to it

Given a system (mobile X-ray system) with: Intermittent failures Unclear problem descriptions like

• “Sometimes the connection between stand and Mobile Viewing Station is lost”

• “Sometimes images are lost” Problems could not be reproduced by functional

tests Root cause of system failure not yet found Log file contains not enough information on

problem


|

Gather the Operational Profile

Log file contains information on usage

Define most important parameters: Switch on / switch

off system Start / stop low dose

fluoroscopy Start / stop high

definition fluoroscopy

Store images

Log file excerpt10-02-10 14:05:34 S INIT > > > > INIT_Main:

Starting application < < < < 10-02-10 14:05:34 S INIT Build by nlybstqvp4-

sys24 on NLYBSTQVP4DT037, 10-02-10 14:05:34 M IPDR DFI Unit Serial

Number: [00-00-10-6B-22-03-01]10-02-10 14:05:34 M IPDR DFI Unit Versions: IN [0x1] PROC [0x1]

CTRL [0x5] DISP [0x1]10-02-10 14:05:34 S IPDR Starting DFI selftest

10-02-10 14:38:55 M SYS USR switch on RIGHT10-02-10 14:38:55 M SYS Xray mode continuous

fluoro10-02-10 14:38:55 M USR START FLUORO RUN RIGHT 10-02-10 14:38:55 M SYS USR switch off at

187683 10-02-10 14:38:56 M USR STOP FLUORO RUN 10-02-10 14:38:56 M FILE Stored [1] new

image(s) on disk10-02-10 14:38:56 M FILE Deleted [0] image(s)

from disk


|


Data from log file


Case#

Start date

dd-mm-yy

Start Time

hh:mm:ss

End Date

dd-mm-yy

End Timehh:mm:s

s

Run Timehh:mm:s

s

Left Fluo#

Right Fluo#

Images Stored

#

Total Fluo time (sec)

Avg Fluo time (sec)

1 10-02-10 14:05:34 10-02-10 14:08:16 0:02:42 0 0 0 0 0

2 10-02-10 14:08:16 10-02-10 14:57:08 0:48:52 2 24 24 15 0.576923077

3 12-02-10 7:12:11 12-02-10 10:27:44 3:15:33 9 74 75 63 0.759036145

38 9-03-10 7:15:20 09-03-10 13:57:47 6:42:27 60 1 18 58 0.950819672

39 10-03-10 7:37:01 10-03-10 7:37:37 0:00:36 0 0 0

Totals 61:27:51 565 480 521 1065

Average 1:34:34 14 12 13 28

|


Interpreting the data determine history classes

Basic profile

A ‘case’ is from system start to system stop Sometimes no fluoroscopy is done at all Most of the times a case consist of fluoroscopy

(LDF, HDF or both)17Real-life test; Real-life situations

History class Start system

Stop system

Make fluoroscopy run(s)

System switched off 1 0 0

System started 0 0.08 0.92

Fluoro Runs completed 0 1 0

|


Add more details for fluoroscopy history classes


History class Start system

Stop system

Make fluoroscopy run(s)

System switched off 1 0 0

System started 0 0.08 0.92

Fluoro Runs completed 0 1 01 – 7 * LDF 8 – 16 * LDF 17 – 45 *

LDF46 – 270 * LDF

Standby 45% 29% 24% 2%

+1 – 7 * HDF 8 – 16 * HDF 17 – 78 *

HDF

Standby 40% 40% 20%

|

Executing the test: Automate

Switch on the systemDecide on the number of fluoroscopy

runs @nr_fluoL = 0 @nr_fluoR = 0 # start throwing the dice for the

left fluoro runs @leftSel = rand() case @leftSel when 0 .. 0.41 @nr_fluoL = 0 when 0.42 .. 0.67 @nr_fluoL = 1 + rand(7) when 0.68 .. 0.84 @nr_fluoL = 8 + rand(9) when 0.85 .. 0.99 @nr_fluoL = 17 + rand(28) else @nr_fluoL = 46 + rand(225) end

# then for the right fluoro runs (= images being stored)

@rightSel = rand() case @rightSel when 0 .. 0.20 @nr_fluoR = 0 when 0.21 .. 0.59 @nr_fluoR = 1 + rand(7) when 0.60 .. 0.78 @nr_fluoR = 8 + rand(9) else @nr_fluoR = 17 + rand(62) end

Perform the selected number of runsSwitch off the system

And start all over again (endless loop)


|

Statistical Usage Test results

System starts showing intermittent failures Finally, test results in a Fatal Error (hard disk

broken)

13 days of test versus 1 month of real-life usage Images stored: 80719 versus 521 Total Fluoroscopy time: 135995 sec versus 1065

sec20Real-life test; Real-life situations

Run Timehh:mm:ss

# Left Fluo

# Right Fluo

# Images Stored

Total Fluo time (sec)

Customer system 61:27:51 565 480 521 1065

Test system 198:19:31 71873 89548 80719 135995

|

Statistical Usage Test results

Problem reproduced Test results have proven the data analysis

Possible improvements on the Operational Profile and/or the test Combine more user data (e.g. more log files, more

customers) Define actual contribution on available data (no

assumption of normal distribution)

Automation is mandatory, e.g. using FitNesse


|

Conclusion

Real-life testing is doing what your customer wants to do Paying attention to thecustomer needs

Real-life testing helps findingissues Help solving thecustomers’ problems


Is your customer satisfied?

Unless you have 100% customer satisfaction…you must improve.Horst Schulze

Software

Real life test; real life situations