John Kent - An Entity Model for Software Testing

4/08/2008 An Entity Model of Software Testing © Simply Testing Ltd

An Entity Model of Software Testing:

A Foundation for the FutureJohn Kent: [email protected]


An Entity Model of Software Testing:

A Foundation for the Future


• Discussions at the Test Retreat• A work in progress• Building Test Management Tools• A Test Entity Model could be used for:

– Software Testing– All types of systems testing


The Need for an Entity Model• Define the language• The Software Testing Cannon

– Myers, etc...– IEEE 829– BSI 7925– ISEB– ISTQB

• For test tools to be built which implement a real model of testing

• How can we train testers without a clear picture of testing?

• Future standards, text books and test tools should be based upon an agreed entity model


• Entity Relationship Diagram - ER Diagram– Entities

• Uses rectangle

– Relationships between entities• Uses ellipse symbol• Many shown as

– Entities have Attributes

Entity Relationship Modeling

Singer

Song

Performs

Song


Sources• Standards

– BSI 7925– IEEE 829

• Training Courses– ISEB– ISTQB

• Glossary• Syllabus

• Experience• Test Retreat discussions• Discussions with testers


Where We Are Now• In many organisations tests look like this:

– Test Script• Test Steps

– Input– Expected Output

Inputs

Expected Outcomes


IEEE 829 Entity Relationship Diagram

• No System Specifications

• No Test Conditions

• Relationships ill-defined


• Tries to describe all possible relationships – not just one way of doing things

• We will build the model by: 1. Using a possible relationship example then…2. Make statements about the relationships


The Entities• Test Entities include:

– Specifications– Test Conditions– Test Cases– Test Scripts– Test Steps– Test Execution Schedule– Test Results

• Not included here – Defects – Test Plans– Test Strategies– Test Coverage Items


Entity: Specification Item• An indivisible, testable part of a specification• Also called Test Basis• Requirement Specification

– User Reqt– Legal Reqt– Performance Reqt– Security Reqt– etc...

• Design/Functional Specification• Attributes

– Description– Risk– Priority– …


Entity: Test Condition• Definition

– ISTQB definition• "An item or event of a component or system that could be verified

by one or more test cases, e.g. a function, transaction, feature, quality attribute, or structural element."

– 7925 definition – not defined– Attributes

• Description• Risk• Priority

• A statement of required system behavior under certain conditions• A refinement of the specification (from a test perspective)• Could be considered to be a specification item


Entity: Test Case• Definition

– ISTQB: “A set of input values, execution preconditions, expected results and execution postconditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement. [After IEEE 610]”

– 7925: “A document providing detailed instructions for the execution of one or more test cases”

– Attributes• Described by IEEE 829• Input• Expected Output• Preconditions• Objective

– Re-use?


Entity: Test Script

• Definition– Also called Test Procedure– ISTQB: Test Procedure – “A document

specifying a sequence of actions for the execution of a test. Also known as test script or manual test script. [After IEEE 829]”

• Consists of Test Steps….• A sequence of test cases?


Entity: Test Step

• Definition– ISTQB: not defined– 7925: not defined

• But they do exist – I’ve seen them!


Entity: Test Execution Schedule

• Tests run sequence• ISTQB: “A scheme for the execution of

test procedures. The test procedures are included in the test execution schedule in their context and in the order in which they are to be executed”.


Entity: Test Result

• ISTQB: Result:– “The consequence/outcome of the execution

of a test. It includes outputs to screens, changes to data, reports, and communication messages sent out.”.

• 7925: Actual Outcome: – “The behavior actually produced when the

object is tested under specified conditions”.


Relationship: Specifications to Test ConditionsExample – an Asset Management System

Spec: Assets

Spec: Add Assets

Spec: Delete Assets

TCond: Test that cannot delete an Asset when used in an active work order

TCond: Test that cannot delete an Asset when used in an active job planA Specification can ‘link’ to

many Test Conditions

TCond: Test that can delete an Asset whenused in an inactive work order

TCond: Test that can delete an Asset when not used in a job plan or work order

TCond: Test that can delete an Asset whenused in an inactive job plan

Together the Specifications and Test Conditions can make up a model of the system under test


Relationship: Specifications to Test Conditions(Cont)

Req Spec: Assets

Req Spec: Add Assets

Req Spec: Delete Assets

TCond: Test that cannot delete an Asset when used in an active work order

TCond: Test that cannot delete an Asset when used in an active job plan


TCond: Test that can delete an Asset whenused in an inactive work order

TCond: Test that can delete an Asset when not used in a job plan or work order

Design Spec: Assets

Design Spec: Delete Asset Menu item

A Test condition could link to many specifications


Relationship: Test Conditions to Test Cases

TCond: Test that can delete an Asset when not used in an active job plan or work order

TCase: Pre-condition: An asset linked to an inactive Job PlanInput: Select ‘Delete Asset’ menuitemExpected Outcome: Asset removed from list


TCond: Test that Asset is removed from the asset list when deleted

The Objective (IEEE 829) attribute in this definition of a Test Case is the linked

test conditions

A Test Case can link to many Test Conditions


Relationship: Test Conditions to Test Cases(cont)

TCase: Pre-condition: An asset linked to an inactive Job PlanInput: Select ‘Delete Asset’ menuitemExpected Outcome: Asset removed from list

TCond: Test that Asset is removed from the asset list when deleted

TCase: Pre-condition: An asset linked to an inactive Work OrderInput: Select ‘Delete Asset’ menuitemExpected Outcome: Asset removed from list

TCase: Pre-condition: An asset not linked to a Work Order or Job PlanInput: Select ‘Delete Asset’ menuitemExpected Outcome: Asset removed from list

A Test Condition can link to many Test Cases


Relationship: Test Cases to Test Scripts

A Test Script can consist of many

Test Cases

A Test Case may appear in many

Test Scripts

Test Script 2:

Test Script 1:

Test Step: 3

Test Case: 2

Test Case: 1

Test Step: 9

Test Case: 2

Test Case: 7

Test Case: 2

This is test case re-use


Test Script 1: Asset Tests

Relationship: Test Scripts to Test Steps

Test Step: 3Pre-condition: An asset linked to inactive JobPInput: Select ‘Delete Asset’ menu itemExpected Outcome: Asset removed from list

Test Step: 2Pre-condition: An active job plan Input: Select ‘Inactivate Asset’ Menu itemExpected Outcome: Asset status=Inactive

Test Step: 1Select an job plan withStatus= ActiveAn Asset linked

A Test Step is a Test Case in a

Test Script


Test Script 2:

Test Script 1:

Relationship: Test Steps to Test Execution Schedules

Test Step: 3

Test Step: 2

Test Step: 1

Test Step: 3

Test Step: 2

Test Step: 1

Test Execution Schedule 1

Script 1 - Test Step 1

Script 2 - Test Step: 2






A Test Execution Schedule can contain

many Test Steps

A Test Step can be used in many Test

Execution Schedules


Test Script2:

Test Script1:


Test Script 1

Test Script 8

Test Script 7


Test Script 2

Test Script 7

Test Script 1

A Test Execution Schedule can contain

many entire Test Scripts

An entire Test Script can be used in many Test Execution Schedules

Relationship: Test Scripts to Test Execution Schedules


Relationship: Test Execution Schedules to Test Results

Test Execution Schedule




Test Results: Run 1

Test Results: Run 2

Test Results: Run 3

A Test Execution Schedule can

appear in many Test Results


The ‘Full’ Test Entity ModelSpecification Item

Test Condition

Test Case

Test Script

Test Step


Test Results


The ‘Full’ Test Entity ModelSpecification Item

Test Condition

Test Case

Test Script

Test Step


Test Results

Test Execution Results must be traceable back through

all of the relationships to the specifications or

requirements


The major difference between the Theory and Real World is the Real World contains time and therefore work we do is incomplete

• It takes time to build the entities• It takes time to build the relationships between entities

• What test entities we build depend on the time we have

• The level of detail in specifications and tests is dependent upon time– Requirements are incomplete– Test conditions are also incomplete


Real World Waterfall Model

Code what you can from specs and make up the rest. Bugs always included

Specify the Syst Design incompletely with errors

Produce Requirements which are incomplete and slightly wrong

There is an over-emphasis on test scripts because it is generally accepted that testers write scripts. Often we could get greater test coverage by producing more test conditions which we could execute directly and not waste time on scripting.

Because Requirements are incomplete we need Test Conditions to describe the system – we produce a list

of incomplete Test Conditions


Additions to the ‘Full’ Test Entity Model

Specification Item

Test Condition

Test Case

Test Script

Test Step


Test Results

We could get greater test coverage by producing more test conditions which we could execute directly.

Writing TCs gives faster test coverage

We have seen that TCs are a form of spec so it would be possible to execute the specifications directly in some test projects


Simplified Test Entity Model 1

Specification Item

Test Condition


Test Results

Test conditions executed directly using testers knowledge to provide missing steps



Test Script

Test Step

Specification Item

Test Condition


Test Results

Entity Model used in Quality Center and

Test Director

Many test teams don’t use this bit

Introduces another addition to the ‘Full’

Model:TCond to Test Script

So they cannot measure test

coverage against requirements

May be difficult to implement full model in a database tool – lots of many to

many relationships



Test Script

Test Step


Test Results

Specification Item

Test Condition

This is where many test organisations are

They may be better off doing this


Summary


Summary

• Full Entity Model Not Completely Defined– Entities not well defined– Relationships not well defined– Attributes not defined – Risk?– Entities left out, Test Coverage, Test Plan

• Once we understand the full entity model we may be able to see more/better ways to test

• In the real world we are likely to use a simplified version of the Entity Model