Testing Internet Services Sudipto Ghosh Sambhrama Mundkur Aditya P. Mathur: PI Ramkumar Natarajan Baskar Sridharan Department of Computer Sciences Purdue

Testing Internet Services

Sudipto GhoshSambhrama MundkurAditya P. Mathur: PIRamkumar NatarajanBaskar Sridharan

Department of Computer Sciences

Purdue University

@ Design2Deploy Monday July 3, 2000

Last updated: June 28, 2000

Test and Management of Internet Services

2

Areas of Concern [1]

Test Methodology Adequacy assessment and test enhancement Load testing/Performance measurement Testing for fault tolerance In-use testing and debugging (dynamic

testing)


3


Management

Uniform interface to a heterogeneous

environment

Generic event and control specification and

processing mechanisms

Variable overhead mechanisms


4

Client/Server

Stub/Skeleton

Request/data

Client/Server

Stub/Skeleton

Request/data

Component Component

ORB ORB

Communication .

Structure of an Internet Service

ComponentORB: Object Request Broker


5

Service Domain

Client

Client Client

An Internet ServiceA component


6

Areas of Concern

Test Methodology Adequacy assessment and test enhancement Testing for fault tolerance Load testing/Performance measurement In-use testing and debugging (dynamic

testing)


7

Elements of a Proposed Test Methodology

Test individual components using traditional black-box and white-box techniques.

For COTS, use interface testing. Prior to deployment, test applications using

Interface Testing. Test for compliance with requirements for FT. Load test an application prior to or during use. While in-use, test and debug an application

using dynamic testing.


8

Areas of Concern


testing)


9

The Coverage Principle

Measuring test adequacy and improving a test set against a sequence of well defined, increasingly strong, coverage domains leads to improved reliability of the system under test.


10

Coverage Domains and Elements

RequirementsClassesFunctionsMutationsExceptionsData-flows

Coverage Domains Coverage Elements


11

Sample Hierarchy

Requirements coverage

Function/method coverage

Statement coverage

Decision coverage

Data-flow coverage

Mutation coverage

Str

engt

hLow

High

Cos

t

Low

High


12

Saturation Effect: Reliability View

FUNCTIONAL, DECISION, DATAFLOWAND MUTATION TESTING PROVIDETEST ADEQUACY CRITERIA.

Reliability

Testing EffortTrue reliability (R)Estimated reliability (R’)Saturation region

Mutation

Dataflow

Decision

Functional

RmRdfRdRf

R’f R’d R’df R’m

tfs tfe tds tde tdfs tdfe tms tfe


13

Component

Interface

Methods:m1, m2, …,mk

Exceptions:e1, e2, …,ek

100% Method Coverage# methods executed# methods defined

100% Exception Coverage# exceptions raised# exceptions defined

100% iMutation Score

# distinguished mutantstotal # imutants - #equivalent imutants

Interface Testing


14

What is Interface Mutation ?

Test Suite T contains Request-A.

Client ServerInterface

Request-A

Response-A

Mutated Interface

Client ServerRequest-A

Response-B


15

Interface Mutation: Experimental Evaluation

Select components Number of components = 3 Number of interfaces = 6 Number of methods = 82 Number of mutants = 220

Seed errors one by one in the components Errors related to:

Portability, data, algorithm, language-specific


16

Characteristics of applications

APP1 Component-3Component-2Component-1

LOC

# Interfaces

# Methods

# Exceptions

# Mutants

1484 1268 2950

2 2 2

27 10 45

1

69

1 1

24 127

0

APP2

LOC

# Interfaces

# Methods

# Exceptions

# Mutants

1893 2799 5559

1 2 2

11 7 9

15

3 4

6 13

ObjectGroupServer

AccountDatabaseManager

ActivityLogger

Client

Component-1 Component-3

Component-2

APP1

Customer Group Service ManagerAccount

Server

ActivityLogger

AccountDatabase Manager Administrator

APP2


17

Control-flow coverage obtained

100%

0

20%

80%

60%

40%

Component-1 Component-2 Component-3

APP1

100%

0

20%

80%

60%

40%

ActivityLogger AccountDBManagerGroupServiceManager

APP2

Block Coverage with TME

Block Coverage with TIM

Decision Cov-erage with TME

Decision Cov-erage with TIM

Covera

ge

Covera

ge


18

100%

0

20%

80%

60%

40%

100%

0

20%

80%

60%

40%

Number of errors revealed


APP1


APP2

Err

ors

Err

ors

Using TME

Using TIM

Using TCF


19

Number of tests required

0

30

0


APP1


APP2

Test

sTest

s| TME |

| TIM |

| TCF |

20

10

10

5


20

Observations [1]

Interface mutation leads to fewer tests that reveal almost as many errors as revealed by statement and decision coverage.

Interface mutation is a scalable alternative to using code coverage.


21

Observations [2]

Reveals programming errors in components. errors in the use of component

interfaces Reveals certain types of deadlocks

41

25

3

Server callback

Client Server

Client Request


22

Areas of Concern


testing)


23

Testing for fault tolerance

Problem: Often error recovery code is not executed

by test inputs How do we know if the fault recovery code

adequately meets the requirements? Solution:

Simulate the occurrence of faults by injecting them

Fault injection testing at the interface Increases coverage of fault recovery code Reveals inadequacies in fault recovery code


24

Interface fault injection

m

Client

Stub Skeleton

Server

ORB

FaultInjected

Simulate effects ofproblems in server,network and ORB


25

Faults suitable for injection

Time delays caused due to Network delays Server crashes or malfunctions Overloaded server Overflowing request buffer in ORB

Unexpected exceptions received by client Server crashes Unexpected value User-defined exceptions raised by the server Implementation object invalid


26

Areas of Concern

Test Methodology Adequacy assessment and test enhancement Testing for fault tolerance Load testing/Performance measurements In-use testing and debugging (dynamic

testing)


27

Load Testing

network network

C1

C2

Server

On: Avg. Latency Effect of: Avg. Load

Clients


28

Server Statistics


29

Client Statistics


30

Low intrusion Selective on-line monitoring Selective on-line filtering To be applied in the

Measurement of the latency and CPU consumption (and others) of individual methods for use in capacity planning

Demonstration of performance improvement techniques

Performance Instrumentation Goals


31

Areas of Concern


testing)


32

Dynamic Testing

Question: How to test an Internet Service while it

is in use? Answer: Use the dynamic testing

procedure.

What is the dynamic testing procedure?


33

Dynamic Testing

FaultyServer groupFaulty

serverTestClient

Ra2

Ra

Client

1

Client Client

Isolatedserver

3


34

Limitations of Dynamic Testing

Test client might generate undesirable actions: Persistent data modification. Irreversible actions.

Application limited to: Closed and well understood domains. Simulated or isolated service

environments.


35


Management


environment





36

Organization of the Service Domain

Why organize ? Efficient and scalable management Personalized management Assignment of individual

responsibilities


37

Dimensions of Organization

Geography

Component

Client


38

Sample Organization


39


Management


environment





40

Monitoring

Why? Resource planning Control Testing and debugging

What? State of individual or a group of

components at different levels of abstraction.


41

Control

Why? Resource allocation Prevention of unauthorized use Testing and debugging

What? Component behavior and location.


42

Monitoring: Strategy

User Event(s)specifies

State Monitor Application Statemonitors

Event detector User specified eventsdetects

Event notification Waiting usernotifies


43

Control: Strategy

User Control action(s), trigger event(s)

specifies

Event notifier Controllersignals

Controller Control action(s)executes


44


Management


environment





45

Observer-listener architecture

O1 O2

C1

O3 O4

C2

L1 M1

O5 O6

C3

O7 O8

C4

L2 M2

Zonal Manager

GUI GUI


46

LLI

MC AR

db

Zonal Manager

LL

CS

LOG

Host 1

C

Architecture of Wabash 2.0 [1]

WabashGUI

Request Client sends a requestto a managed object

LL determines whether request can be passed or not

If the request is allowed, LL forwards it to the CORBA Server after time-stamping it

Response

LL stores information about the request, records it in a log, sends the response back to client

LL gets the responsefrom the CORBA Server

If the request is not allowed,LL throws exception and doesnot forward request to the CORBA Server

GUI requests data fromthe Zonal Manager

Zonal Manager requests datafrom corresponding LL

LL returns datato Zonal Manager

Zonal Manager returnsdata collected from LLto GUI


47

LLI

MC AR

db

Zonal Manager

LL CS

LOG

Host 1

C

WabashGUI

Request

Architecture of Wabash 2.0 [2]

Manager sends command(‘ANY’, DENY, OBJECT_A)

Command is forwarded tocorresponding LL for OBJECT_A

Client sends requestto OBJECT_A

LL determines that requesthas to be denied

LL does not forwardrequest to CSThrows exception backto client


48

Architecture of Wabash 3.0


49

Ongoing Research [1]

Non-intrusive procedures for dynamic testing.

Generalized event-control model and its implementation.

Implementation of the unified architecture to assist with the management of JMX, JINI, and CORBA objects.

Light-version of Wabash for SmartHome management.


50

Ongoing Research [2]

Automatic generation of test inputs.

Test capture and replay.


51

The Wabash Project: History

Progress:August 1998:

Wabash project launched.

August 1999 TDS 1.1 available to SERC affiliates.

August 2000 Wabash 2.0 available to SERC affiliates. Experiments to assess goodness of proposed

interface testing criteria completed.

December 2000 Uniform interface for Jini/JMX/CORBA objects.

Documents

Testing Internet Services Sudipto Ghosh Sambhrama Mundkur Aditya P. Mathur: PI Ramkumar Natarajan Baskar Sridharan Department of Computer Sciences Purdue