Measuring Test Execution Complexity Eduardo Aranha and Paulo Borba Informatics Center Federal University of Pernambuco Recife, PE, Brazil {ehsa, phmb}@cin.ufpe.br

Measuring Test Execution Complexity

Eduardo Aranha and Paulo Borba

Informatics CenterFederal University of Pernambuco

Recife, PE, Brazil

{ehsa, phmb}@cin.ufpe.br

Mobile Devices R&D Motorola Industrial LtdaJaguariuna, SP – Brazil

PROMISE 2006

Motorola Proprietary, MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other

product or service names are the property of their respective owners. © Motorola, Inc. 2004. Mobile Devices Software Group

Page 2

Software Testing

• Companies that release products with poor quality may quickly lose its clients

• A usual activity performed to ensure quality is software testing

Test team

Developers teams

Developer team



Page 3

Research Context

Software Requirements

Test Cases

Requirements and Code Coverage

Test Execution Effort Estimates

Test Generation

EffortEstimation

CoverageAnalysis

Test Case Selection

Test CaseSelection

Test Team and Resource Planning

PlanningQuality Model

This Research

Others Motorola Brazil Test Center Researches

Some Others Motorola Researches

Motorola Brazil TestCenter Research Project



Page 4

Test Specification Language

• Controlled natural language (CNL)– Subset of English with restricted grammar and vocabulary– Domain specific language (e.g., mobile phone application)– Sentences written in a more concise and standard way– Make easer to understand and to have text processing

Example of test procedure:

Step Description Expected Results

1 Start the message center. The phone is in message center.

2 Select the new message option. The phone is in message composer.

3Insert a recipient address into the recipients field.

The recipients field is filled.

4Insert a SMS content into the message body.

The message body is populated.

5 Send the message.The send message transient is displayed. The message is sent.



Page 5

Measuring Test Execution Complexity of a Test Case

• For each single test action (step)– Evaluate execution complexity levels– Rate execution complexity levels– Add points to the whole test execution complexity

Example:Step Description Expected Results

1 Start the message center. The phone is in message center.

2 Select the new message option. The phone is in message composer.

3Insert a recipient address into the recipients field.

The recipients field is filled.

4Insert a SMS content into the message body.

The message body is populated.

5 Send the message.The send message transient is displayed. The message is sent.

100

Points Complexity LevelCharactistic

78Total

.........

20LowNFC2

0N/ANFC1

.........

10HighFC2

30LowFC1

2

Total: 630 execution points

3

1


Total

......

LowNFC2

N/ANFC1

......

HighFC2

LowFC1


Total

...

NFC2

NFC1

...

FC2

FC1



Page 6

Step 1 – Evaluating execution complexity levels

• Knowledge base for the execution complexity levels of the test action – Verb and optionally its arguments mapped into complexity levels

• If it is a new action:– Evaluate it according to the exercised system functional and non

functional characteristics– Record evaluation in the knowledge base

FC2 Average size of date inputs.

Less than 30 30 to 100 More than 100

Low Medium High

NFC2 Use of Bluetooth.

File transfer, print messages, use

headset

Application data synchronization

30 50



Page 7

Step 2 and 3

• Step 2: Rating execution complexity levels – Search the knowledge base for the values assigned for each

execution complexity levels of each characteristic

• Step 3: Assigning the value for the test case– The execution complexity of a test case is calculated by

summing the execution points of each of its test steps

FC2 Average size of date inputs.

Low Medium High

15 60 150



Page 8

Using Test Execution Complexity for Effort Estimation

• Test productivity– The proportion between execution points and time spent for

executing each test

– For example, a test with 600 execution points executed in 5 minutes gives a productivity of 120 execution points / minute

– A new test case with 160 execution points can be estimated to be executed in approximately 1 minute and 20 seconds

• Risk factors– Environment conditions, team experience, the use of tools, the

reuse of test setups and other factors must be considered

– Effort multipliers



Page 9

Method Validation

• Initial experiment– We observed the relation between execution times and the

measured complexity of test cases– Qualitative analyses based on historical data with different

environment conditions

• After submitting final paper version, we executed and analyzed 33 tests in a controlled environment– We extended the characteristics using a Delphi panel– Improvements:

Test MMRE PRED(20) PRED(30)

136.75% 100.00% 37.50%

236.12% 33.33% 25.00%

317.19% 50.00% 0.00%



Page 10

Conclusions

• New method for measuring test execution complexity

• Applicable for new or existing tests

• CNL improves specifications and supports automation– Knowledge base reduces estimation effort

• Some drawbacks– Effort to create the knowledge base– CNL, functional and non functional characteristics are dependent of

application domain

• More experiments to run – ANOVA, covariance– Estimation accuracy, regression analysis, Bayesian calibration, etc.



Page 11

Questions?



Page 12

Functional Characteristics

ID DescriptionComplexity Level / Guidelines / Rates

Low Average High

1Average number of navigations between screens.

Up to 5 More than 5

9 13

2 Average size of date inputs.Less than 30 30 to 100 More than 100

15 60 150

3 Software configuration.

e-mail or IM account configuration

40

4 File manipulation.

save filesMove files from different memories

types or transfer files though network

20 50

5 List manipulation.

search list entries, delete entries add new entries

15 20

6 Multimedia manipulation.

Standard sound/picture/animation

manipulationcamera, emoticon

specific sound/picture/animation, edit picture

21 26 38

7 Server access type.

Standard Authenticated

40 60

8 Type of screen items to be verified.Valid characters Invalid characters Number of pixels

18 35 50



Page 13

Non Functional Characteristics

ID DescriptionComplexity Level / Guidelines / Rates

Low Average High

1 Application delay.

Wait a transient message

10

2 Use of Bluetooth.

File transfer, print messages, use headset

Application data synchronization

30 50

3 Use of network.Sending IM messages

or short e-mails or short messages

Sending large messages or large e-

mails

Documents

Measuring Test Execution Complexity Eduardo Aranha and Paulo Borba Informatics Center Federal University of Pernambuco Recife, PE, Brazil {ehsa, phmb}@cin.ufpe.br