System Integration testing

1

Overview• Testing• System Integration Testing• Integration testing

▫Big bang▫Bottom up▫Top down▫Sandwich

• System testing▫Functional▫Performance

• Acceptance testing• Case study

2

Testing

What is the overall goal of testing? What claims can we make when testing "passes" or "fails" ? Can we prove that our code has no bugs?

3

Testing: systematic approach to find the errors in system (to "falsify" system)

accomplished by exercising defects in the system and revealing problems failed test: an error was demonstrated passed test: no error was found, so far

not used to show absence of errors in software does not directly reveal the actual bugs in the code

System Integration Testing

• (S.I.T.) is the testing of the sub-systems, as a whole, to ensure that they work as a system.

• verifies the proper execution of software components and proper interfacing between components within the solution. 

• For eg. In case of lab thermostat changes in the environment can affect the performance of a system.

4

Integration Testing [i]

• The entire system is viewed as a collection of subsystems (sets of classes) determined during the system and object design.

• Goal: Test all interfaces between subsystems and the interaction of subsystems.

• The Integration testing strategy determines the order in which the subsystems are selected for testing and integration.

5

Why do we do integration testing?• Unit tests only test the unit in isolation

• Many failures result from faults in the interaction of subsystems

• Often many Off-the-shelf components are used that cannot be unit tested

• Without integration testing the system test will be very time consuming

• Failures that are not discovered in integration testing will be discovered after the system is deployed and can be very expensive.

6

Stubs and drivers

• Driver:▫A component, that calls the TestedUnit▫Controls the test cases

• Stub:▫A component, the TestedUnit

depends on▫Partial implementation▫Returns fake values.

Driver

TestedUnit

Stub

7

Example: A 3-Layer-Design

Layer I

Layer II

Layer III

A

alculator

C

BinaryFileStorage

E

XMLFileStorage

F

CurrencyDataBase

G

CurrencyConverter

D

DataModel

B

A

C

E F G

DB

SpreadSheetView

BinaryFileStorage

EntityModel

A

E F

CurrencyDataBase

G

CurrencyConverter

DB

Calculator

C

XMLFileStorage

(Spreadsheet)8

Integration Techniques [iii]

Common approaches to perform integration testing:• Top-down (from higher levels no test drivers are needed)• Bottom-up (from lower levels No test stubs necessary)• Sandwich (combination of bottom-up and top-down no test

stubs and drivers needed)• Big-bang (all components together)

Pre-requisite A module must be available to be integrated

A module is said to available for combining with other modules when the module’s check-in request form is ready

9

Top-down Testing Strategy

• Test the top layer or the controlling subsystem first• Then combine all the subsystems that are called by the tested

subsystems and test the resulting collection of subsystems• Do this until all subsystems are incorporated into the test• Special program is needed to do the testing, Test stub :

▫A program or a method that simulates the activity of a missing subsystem by answering to the calling sequence of the calling subsystem and returning back fake data.

10

Top-down Integration

Test A, B, C, D,

E, F, G

All LayersLayer I + II

Test A, B, C, D

Layer I

Test A

A

E F

B C D

G

11

Pros and Cons of top-down integration testing

• Test cases can be defined in terms of the functionality of the system (functional requirements)

• Writing stubs can be difficult: Stubs must allow all possible conditions to be tested.

• Possibly a very large number of stubs may be required, especially if the lowest level of the system contains many methods.

• One solution to avoid too many stubs: Modified top-down testing strategy▫Test each layer of the system decomposition individually

before merging the layers ▫Disadvantage of modified top-down testing: Both, stubs and

drivers are needed

12

Bottom-up Testing Strategy

• The subsystems in the lowest layer of the call hierarchy are tested individually.

• Then the next subsystems are tested that call the previously tested subsystems.

• This is repeated until all subsystems are included.• Drivers are needed.

13

A

C

E F G

DB

Bottom-up IntegrationA

Test A, B, C, D,

E, F, G

ETest E

F

Test F

B

Test B, E, F

C

Test C

D

Test D,G

G

Test G

14

Pros and Cons of Bottom-Up Integration Testing

•Con:▫Tests the most important subsystem (user interface) last.▫Drivers needed.

•Pro:▫No stubs needed▫Useful for integration testing of the following systems:

Object-oriented systems Real-time systems Systems with strict performance requirements.

15

Sandwich Testing Strategy• Combines top-down strategy with bottom-up strategy• The system is view as having three layers

▫A target layer in the middle▫A layer above the target▫A layer below the target▫Testing converges at the target layer

• How do you select the target layer if there are more than 3 layers?▫Heuristic: Try to minimize the number of stubs and drivers

16

Sandwich Testing Strategy

Test A, B, C, D,

E, F, GTest B, E, F

Test D,G

Test A

Test E

Test F

Test G

Test A,B,C, D

A

E F

B C D

G

17

Pros and Cons of Sandwich Testing

• Top and Bottom Layer Tests can be done in parallel

• Does not test the individual subsystems thoroughly before integration

• Solution: Modified sandwich testing strategy

18

Big Bang• Bring all components together all at once. All

interfaces tested in one go.• Entry criteria – all components have passed unit testing• Exit criteria – Test suite passes

19

A

C

E F G

DB

Big-Bang ApproachTest A

Test B

Test G

Test F

Test E

Test C

Test DTest

A, B, C, D,E, F, G

20

Steps in Integration-Testing [v]

1. Based on the integration strategy, select a component to be tested. Unit test all the classes in the component.

2. Put selected component together; do any preliminary fix-up necessary to make the integration test operational (drivers, stubs)

3. Do functional testing: Define test cases that exercise all uses cases with the selected component

1. Based on the integration strategy, select a component to be tested. Unit test all the classes in the component.

2. Put selected component together; do any preliminary fix-up necessary to make the integration test operational (drivers, stubs)

3. Do functional testing: Define test cases that exercise all uses cases with the selected component

4. Do structural testing: Define test cases that exercise the selected component

5. Execute performance tests6. Keep records of the test cases

and testing activities.7. Repeat steps 1 to 7 until the

full system is tested.

The primary goal of integration testing is to identify errors in the (current) component configuration.

4. Do structural testing: Define test cases that exercise the selected component

5. Execute performance tests6. Keep records of the test cases

and testing activities.7. Repeat steps 1 to 7 until the

full system is tested.

The primary goal of integration testing is to identify errors in the (current) component configuration.

.

21

System Testing [ii]

•Functional Testing▫Validates functional requirements

•Performance Testing▫Validates non-functional requirements

•Acceptance Testing▫Validates clients expectations

22

.

Functional Testing

Goal: Test functionality of system•Test cases are designed from the requirements analysis

document (better: user manual) and centered around requirements and key functions (use cases).

•The system is treated as black box.•Unit test cases can be reused, but new test cases have to

be developed as well.

23

Performance Testing

Goal: Try to violate non-functional requirements

•Test how the system behaves when overloaded. •Try unusual orders of execution •Check the system’s response to large volumes of data•What is the amount of time spent in different use

cases?

24

Types of Performance Testing• Stress Testing

▫ Stress limits of system

• Volume testing▫ Test what happens if large

amounts of data are handled

• Configuration testing▫ Test the various software and

hardware configurations

• Compatibility test▫ Test backward compatibility with

existing systems

• Timing testing▫ Evaluate response times and time

to perform a function

• Security testing▫ Try to violate security

requirements

• Environmental test▫ Test tolerances for heat, humidity,

motion

• Quality testing▫ Test reliability, maintain- ability &

availability

• Recovery testing▫ Test system’s response to presence

of errors or loss of data

• Human factors testing▫ Test with end users.

25

Acceptance Testing• Goal: Demonstrate system is

ready for operational use

▫ Choice of tests is made by client

▫ Many tests can be taken from integration testing

▫ Acceptance test is performed by the client, not by the developer.

• Alpha test:▫ Client uses the software at

the developer’s environment.

▫ Software used in a controlled setting, with the developer always ready to fix bugs.

• Beta test:▫ Conducted at client’s

environment (developer is not present)

▫ Software gets a realistic workout in target environ- ment

26

The 4 Testing Steps [iv]

1. Select what has to be tested▫ Analysis: Completeness of

requirements▫ Design: Cohesion▫ Implementation: Source code

2. Decide how the testing is done▫ Review or code inspection▫ Proofs (Design by Contract)▫ Black-box, white box, ▫ Select integration testing

strategy (big bang, bottom up, top down, sandwich)

1. Select what has to be tested▫ Analysis: Completeness of

requirements▫ Design: Cohesion▫ Implementation: Source code

2. Decide how the testing is done▫ Review or code inspection▫ Proofs (Design by Contract)▫ Black-box, white box, ▫ Select integration testing

strategy (big bang, bottom up, top down, sandwich)

3. Develop test cases▫ A test case is a set of test

data or situations that will be used to exercise the unit (class, subsystem, system) being tested or about the attribute being measured

4. Create the test oracle▫ An oracle contains the

predicted results for a set of test cases

▫ The test oracle has to be written down before the actual testing takes place.

3. Develop test cases▫ A test case is a set of test

data or situations that will be used to exercise the unit (class, subsystem, system) being tested or about the attribute being measured

4. Create the test oracle▫ An oracle contains the

predicted results for a set of test cases

▫ The test oracle has to be written down before the actual testing takes place.

27

Case Study [vi]

28

“Improved Software QA Testing with TESTCo”

“Improved Software QA Testing with TESTCo”

Problems of Journyx When Journyx originally released their software solution, Journyx

Timesheet, features were added week-in and week-out without any serious testing considerations. Problems are:

1. The flow of bad features adding to their products without understanding the impact of those features .

2. By regression testing ,the system slows down the new version to release and sometimes the Customers finding severe defects.

3. They hire an expert to document their Software QA testing procedures and for developing manual test scripts , but still finds difficulty in getting desired results.

29

Solutions

• The company then hires an expert team from TESTCo , which gives them suggestions to improve the Software QA testing.

• TESTCo expanded and standardized Journyx’s software QA testing coverage which also helps in speeding up their development process and anyone with the minimal training can run a test.

• TESTCo also automated a large portion of test coverage due to the number of platforms used by their customers.

• Due to the automated testing coverage, daily basis reports of continuous regression testing and defects can be generated which results in easier finding the desired results and also ease the development of processes.

30

Results In terms of testing : Cutting of testing cycles from 96 business days to 32 business

days.New platforms can be easily added and tested within 12 hrs

which helps in future development.By doing regression testing, they find zero critical defects

when adding new features. In terms of Budget :They’ve dodged a lot of hidden costs.Any person can easily be trained by TESTCo’s team by

paying them within three months for the future improvement of company.

31

References

i. http://www.exforsys.com/tutorials/testing/integration-testing-whywhathow.html

ii. http://www.exforsys.com/tutorials/testing/system-testing-whywhathow.html

iii. http://softwaretestingguide.blogspot.com/iv. http://www.softwaretestingstuff.com/2008/12/integration-

testing-four-step-procedure.htmlv. http://forum.onestoptesting.com/forum_posts.asp?TID=1494vi. http://www.testco.com/case-studies/journyx2-case-

study_en.html

32

THANK YOU

33