Testware Hierarchy for Test Automation

1

Testware Hierarchy for

Test Automation

Gregory Solovey

The proposed process allows to perform test automation in parallel with coding, and can be the base for continuous integration and agile testing

2

Developer : unit test

Team: feature test

Project: regression & sanity test

Product: system & load test

Technology: interoperability test

In the traditional approach automated tests are added during subsequent test process phases

In the proposed approach automated tests are reused while transitioning to the subsequent test process phases

Test Automation

CREATION

EVOLUTION

3

In the traditional approach test is sequential (formal relay)

In the proposed approach test is almost parallel(multi-threaded, with different priorities)

Agile Testing

Requirements Design Coding Unit test

Scenarios

Test plan Testing

Testware Testing

4

Processes

Provide decomposition/

abstraction

Design Tests

Build Harness

Automate Testing

There Is Nothing New Under the Sun

Divide

and

rule

It is

eas

ier t

o

perc

eive

erro

r

than

to fi

nd tr

uth

Giv

e m

e a

leve

r

and

a fu

lcru

m a

nd I

shal

l mov

e th

e

wor

ld

He who

lives

mus

t be

prep

ared

for

chan

ges

5

Processes

6

Processes


abstraction

Define hierarchy

Present as a formal model

Design Test

Order error set

Cover all faults

Build Harness Provide

controllability/ observability

Automate Testing

Write scripts in test language

Separate business functions from

implementation

Architecture/ Requirements docs

Test docs

Design docs

Testware

Processes

7

Hierarchy

8

Processes


abstraction

Define hierarchy

Present as a formal model

Design Test

Order error set

Cover all faults

Build Harness Provide

controllability/ observability

Automate Testing

Write scripts n test language

Separate business functions from

implementation

•Systems hierarchy

•Models hierarchy

•Errors hierarchy

•Tests hierarchy

•Harness elements hierarchy

•Report hierarchy

•Testware hierarchy

Order is Heaven's first law

9

04/13/2023 10

Process: Provide object decomposition/abstraction Sub-process: Define object hierarchyPurpose: Simplify the test design process

• As a rule of thumb, the size of a subsystem to be tested should be in the order of a hundred elements (objects, states, messages)

• Represent the system hierarchically, from a high level description to a less abstract one, until the desired level of detail is achieved

• Vertical decomposition is achieved by breaking down an element from black box to white box; from functional to structural model

• Horizontal decomposition is achieved by ensuring the independence of the model's elements.

Algorithms

Use cases

Sub-system

System

Object Test Harness Auto

04/13/2023 11


• Provides a natural way to divide the work• Achieves the fastest way to find most critical

errors by applying tests to the highest level of object abstraction first

• Defines the boundary between system, feature, unit test

Algorithms

Use cases

Sub-system

System


04/13/2023 12


Algorithms

Use cases

Sub-system

System

Nowadays the systems’ architecture provides the right level of decomposition:

In the vertical view the direction is to separate the business and implementation details, allowing for independent multi-layer development, where the lower level layers are unaware of the higher layers

In the horizontal view the direction is towards object reuse (component based development), with the ability to assemble the objects in various contexts and access them via APIs

Hierarchy in SW architecture: (application (middleware (operating system(firmware (drivers))))Hierarchy in HW models: (processors (registers (flip-flops (gates)))Hierarchy in Protocol models: (SIP(TCP(IP (PPP(DS)))))


04/13/2023 13


Algorithms

Use cases

Sub-system

System

For systems where this condition is not satisfied the tester has to provide the decomposition, document independent objects, review assumptions with systems engineers, estimate risk of such assumptions.

The management has to decide whether to accept the risk that test is incomplete or request architectural changes to satisfy the test requests.


04/13/2023 14

Business function Model Test

Process: Provide object decomposition/abstraction Sub-process: Present as a formal modelPurpose: Formalize the test design process

• If the architecture/requirements are presented as a set of business functions, then only subject matter experts can define tests, based on their expertise. There are no formal rules on how to build tests from business-oriented requirements; therefore, it cannot be proved that the tests are complete and consistent.

• If the architecture/requirements are presented as a set of formal software engineering models, then testers can use well-known test design approaches to cover all model errors.


04/13/2023 15

Business function Model Test

Process: Provide object decomposition/abstraction Sub-process: Present as a formal modelPurpose: Formalize the test design process

Data structure

Condition; function

Structural: algorithm; ladder diagram

Functional: State machine; Syntax; Object-oriented

Compound: Instruction set


For systems where requirements are described as business functions the tester has to transform the plain, narrative text into formal statements (models analogies to UML), and have the result reviewed for consistency

04/13/2023 16

Business Model Errors Test

• A test cannot prove the correctness of an object, but can identify its errors. A test is a set of object stimuli that forces the object to produce different responses in the presence or absence of errors.

• Errors are possible distortions of the object-to-test’s model: its elements, structure or behavior.

Process: Design testSub-process: Order the error setPurpose: Increase error resolution


04/13/2023 17

Process: Design testSub-process: Order the error setPurpose: Increase error resolution

4

1

32

54

1

3

2

regression test

each test case attempts to cover as many errors as possible (to have fewest cases)

new feature test

each test case attempts to cover as few errors as possible (to reach highest error resolution)


04/13/2023 18

Model Errors Method Test

Process: Design test Sub-process: Cover all faultsPurpose: Guarantee the coverage with a minimal test set

Model MethodElementary: Variable; Condition; Function

Boundary analysis; Graph coverage; Value changes

Structural: Data/control flow; Algorithm

Nodes/ Branches/ Path coverage

Functional: State machine; Syntax; Object-oriented

States and transition functions coverage; Rules coverage; Properties, methods coverage

Compound: Instruction set Sensitive Path


04/13/2023 19

a = b or c;f = a & d;if (f) then {} else {}

D = {a1, a0, b1, b0, c1, c0 , d1, d0, f1, f0 }

b

cd

bcd ExpectedResult

Errors

101 0 a0 b0 d0 f0

011 0 c0

001 1 a1 b1 c1 f1

110 1 d1

bcd ExpectedResult

errors

000 1 f0

111 0 a0 d0 f0

001 1 a1 b1 c1

110 1 d1

011 0 c0

101 0 b0

Regression test

Feature test

f1

a0 d0 f0

a1 b1 c1

c0 b0

d1

Graph model

Error set

Error tree

Program Object Test Harness Auto

19

04/13/2023 20

Process: Build the test harness Sub-process: Provide controllability/observabilityPurpose: Guarantee the test execution

execute

compare

control

observe

Object to test

A test harness is a set of “instruments” that provides the ability to execute test cases:

• CLI commands to access APIs or test functions• GUI hidden objects to access database or send transactions


subsystem

21

Recommendation for CLI commands: • Provide routing to a particular CLI command, using UNIX-like syntax

(“cd”, “ls”, “help”, “show”)• Print the states and values of the object or group of objects in TLV

format • Access all object APIs: create, activate, discover, delete, send, etc• Report errors with the keyword “ERROR”• Redirect system outcoming messages to CLI interface • Do not perform any checks other than syntax at the CLI level• Return “OK/ ERROR” results from the command

Recommendation for GUI objects: • Use unique names for application objects • Use naming conventions• Use automation-friendly controls

Process: Build the test harness Sub-process: Apply standardsPurpose: Simplify automation


Process: Automate testingSub-process: Separate business function from implementation detailsPurpose: Provide testware maintenance


Developers and testers have different preferences for test tools: • Developers prefer to use open source tools (to be able to modify

them) and write test cases in a programming language (C, VB, C#, PERL, EXPECT)

• Testers feel comfortable using production tools (access to technical support) and write test cases in specialized test languages (test framework)

Programming language Test language

variables, arrays, hashes TS,UC,TC,TA w/ parameters

if, case compare

for, while, until loop for TS,UC,TC,TA

function, subroutine, procedure TC {setup, test, compare, reset}

algorithm unconditional sequence22

Process: Automate testingSub-process: Write script in a test language Purpose: Simplify debugging

Script: capture & playback

Test Set: subsystem level

Use Case: requirements level

Test Case: condition level

Test Action: transaction level


Sample framework shows various test object levels: (TS (UC (TC (TA))))

23

Process: Automate testingSub-process: Write script in a test language Purpose: Guarantee reporting diversity


level verdict report

TS subsystem metrics product management

UC feature full/partial/no functionality project management

TC test case pass/ fail developer/tester

TA transaction ok/ error/ timeout developer/tester

24

25000 20 5000 200 250 250 50 50SUM UNIQUE Test

TS 2

UC1

TC 1

TA 1 .TA 2

… TC q

TA 1 TA 2

… UC n

TC 1

TA 1 TA 2

… TC w

TA 1 TA 2

… TS n

UC 1

TC 1

TA 1 TA 2

… TC r

TA 1 …. TA 2

… UC m


TS

•XML config file

UC

•Test management system

TC

•Libraries/ Scripts

TA

•Test Language


Sample testware hierarchy for an LTE embedded middleware system:• XML configuration file contains all test sets , which are invoked

based on changes to subsystems detected during the build process (for continuous integration)

• The test management system contains the use cases• The libraries contain the test cases (set-execute-compare-reset

phases)• The test language describes possible interactions with the object-

to-test

25


Test file call Test file call

Test file

Test case

Compare

Action

Action

Action

ActionAction

…

…

Test phase

Reset phase

Test caseTest case

Test case== definition part ========================

Use case

Test case callTest case callTest case call

Use case


Use case


== control part ==============================

TEST LIBRARIES

TEST CONTAINERS

TEST SCRIPTS


26

Software

Testware Hierarchy for Test Automation