The Core of Testing – Dynamic Testing Process – According to ISO 29119 with Anne Mette Hass

The Core of Testing- Dynamic Testing Process -

According to ISO 29119

EuroSTAR Webinar24th September 2014

Anne Mette [email protected]

Please test this!

Test basisTest item

The Example Test Item

Evolutionary development

Development of IT systems

RDCI

RDCI

RDCI

RDCI

RDCI

RDCI

RDCI

Agile development

Sequential development

Testing of IT systems

Test is a support process for development

Test sub-processes are designed according to development phases and requirements

Definitions of test

In Denmark we have a saying: ”Charished child has many names.”

Testing has many definitions, 2 of them are:

• ISO 29119: ”set of activities conducted to facilitate discovery and/or evaluation of properties of one or more test items”

• Lee Copeland: ”Testing is comparing what you have to what you expected to get”

Classification xxxx Author xxxx Approved by xxxx Version xx6

Test basis is indispensible

Test basis may for example be: • Requirements • User stories• Use cases• Functional descriptions• Behaviour descriptions• Rules • Standards• Design at various levels• Experience


NEVER, EVER the code

Test Management

Testing must be planned, monitored and controlled like any other activity in the world


Test management

Completion

Planning Monitoring and control

Be prepared -Be prepared -

Basic Test Types

Types can be Mixed

A test sub-process can for example be • a component test• a functional system test• a performance test

Dynamic Test

This webinar will concentrate on the dynamic testing process , especially the

Test design and implementation

activity

Paul Gerrard’s New Testing Model


Test Design and Implementation

Feature Sets

- or divide and conquer

A feature set is a logical subset of the test item

• for a component test we might have just one feature set

• for a system test we might have dozens

Examples

Feature set 1: Use CasesFeature set 1.1 LoginFeature set 1.2 Earn PointsFeature set 1.3 Spend PointsFeature set 1.4 Benefits


Feature set 2: NavigationFeature set 2.1 MenusFeature set 2.2 Keys

Feature set 3: Error HandlingFeature set 3.1 TextsFeature set 3.2 Severity

Test Condition

A test condition is a testable aspect of a test item such as a verifiable

• function

• transaction

• feature

• quality attribute

• structural element

There may be one or many for each feature set

Description of Member Levels

“There are 3 member levels: Basis, Silver, Gold. Your member level is determined by the number of Basis Points you earn within your personal 12 month period. You will automatically be upgraded to Silver Member if you earn 20.000 Basis Points during your earning period. If you earn 50.000 Basis Points in the period you a become Gold Member. The earning period runs for 12 months from the first day of joining.”

Now test

that!

Derived Test Conditions

• TCO-1.1: A member starts as a Basis member.

• TCO-1.2: A member will automatically be upgraded to Silver Member if he/she earns 20.000 Basis Points during an earning period. If he/she earns 50.000 Basis Points in a period he/she becomes Gold Member.

• TCO-1.3: The earning period runs for 12 months from the first day of joining.

This activity usually creates lots of questions.

Test Coverage Items

A test coverage item is an attribute or combination of attributes that

is derived from one or more test conditions

by using a test case design technique.

Each test case design techniques provides its own coverage element

Derived coverage item

• TCO-1.2: A member will automatically be upgraded to Silver Member if he/she earns 20.000 Basis Points during an earning period. If he/she earns 50.000 Basis Points in a period he/she becomes Gold Member.

• CI-1.2-1 0 <= earned points < 20.000

• CI-1.2-2 20.000 <= earned points <50.000

• CI-1.2-3 50.000 <= earned points


Specification-based Test CD Techniques

• Equivalence partitioning• Classification tree method• Boundary value analysis• Domain analysis• Syntax testing• Combinatorial testing• Decision tables• Cause-effect graph• State transition testing• Scenario testing• Random testing

• Partition• Leaf• 2/3 boundary values• In- and Out-points• None• Combination • Combination• Combination• Chow’s n switch• Flow• None

Structure-based Test CD Techniques

• Statement testing• Decision testing• Branch condition testing• Branch condition

combination testing• Modified condition

decision coverage testing• Path testing• Intercomponent testing

• Statement• Decision outcome• Branch• Branch condition

outcome• Modified condition

• Path• Call

Experience-based Test CD Techniques

• Error guessing• Checklist-based• Exploratory testing• Attacks

• Defect taxonomies• Fault injection and

mutation

• None• List item• None• Registered attack

• Entry• None

Test Cases

Test cases are used to guide the dynamic test.

A test case is a set of input and the test item’s expected reaction.

Test Environment


Detailed Test Environment Requirements ID Description Responsible

ER1 Hardware: Three PCs Production

ER2 Software: MS Windows 7 or later Production

…

ERn Security controls are identified in the Corporation Security Protocol.

Security manager

ERm Refer to the Test Plan for relevant testing tools.

Production and development.

…

Detailed Test Data Requirements ID Description Responsible Reset A/D

DR1 At least 20 members for various countries in the database.

Production Yes A

DR2 An ordinary user and a system administration user.

Production N/A D

…

DRn Table with 5 different trips with different points.

Production N/A A

DRm Table with membership status and necessary point to obtain each.

Production NA A

…

Test Data

Test set and test procedure

• A test set is a collection of test cases selected for execution for a specific purpose

• A test procedure is a recipe for the test execution where the test cases are sorted in a reasonable order

Test Procedure


Test procedure: P11 (3)

Case Input Expected output Actual output

3.6 (17)

Enter… The sum …

3.8

(18)

Enter… The value..

Stop and wrap up: Run the “Clean 1” script.

The information about which test procedures have been executed and with

Objective: This test procedure tests …

Priority: 2

Start up: The form F1 must be current.

Relationships to other: None

Expected duration: 15 min.

Execution time: Initials:

System: Result:

Log result

Log when Log who

Identify item etc. Log overall test result

Test Execution - The moment of thruth!

• Don’t make a false start – check the start criteria

• Follow the test procedures, because:

• actual time spent should be comparable with estimates

• progress should be comparable with the plan

• the tests should be exactly repeatable for retesting and regression testing

• a complete audit of the test may be required

Identifying Failures

• Be very careful when comparing the expected result with the actual result

• in order not to miss failures (called false-positives)

• report correct behavior as failures (called false-negatives)

Actual result = Expected result => Test result = Passed

Actual result = Expected result => Test result = Failed

3

Test Execution Logging

Logging • is essential• is often done directly in the procedure • can also be done in a tool

Test procedure: P11 (3)

Case Input Expected output Actual output

3.6 (17)

Enter… The sum …

3.8

(18)

Enter… The value..

Stop and wrap up: Run the “Clean 1” script.

The information about which test procedures have been executed and with

Log result

√

I-5342

The End

Commercial


Remember

• Test is difficult

• Test requires overview

• Test requires creativity

• Test requires systematic work

• Test requires imagination

• Test requires courage

• Test is fun11-04-202334

Technology

The Core of Testing – Dynamic Testing Process – According to ISO 29119 with Anne Mette Hass