Review and Test Plan

8/12/2019 Review and Test Plan

http://slidepdf.com/reader/full/review-and-test-plan 1/15

Template for IDA Project (Project Id)

Template for specific development (Contract Id)

Review and Test PlanIssue 1



Review and Test Plan Page iiIDA-MS-RTPIssue 1

TABLE OF CONTENTS

0 PREFACE...................................................................................................................... 10.1 Purpose of this document............................................................................................. 10.2 Use of this document........................................................................................... ......... 10.3 Overview........................................................................................................................ 20.4 estin! within "#A................................................................................................ ....... 20.$ Purpose of the Review %nd est P&%n.......................................................................... 30.' Evo&ution of this P&%n................................................................................................... 3

1 "( RO#UC "O(........................................................................................................ 41.1 Purpose.......................................................................................................................... 41.2 App&ic%)&e %nd Reference #ocuments........................................................................ 41.3 #efinitions..................................................................................................................... $

2 *ER"F"CA"O( O*ER*"E+................................................................................... '2.1 Or!%nis%tion , Processes "nvo&ved............................................................................. '2.2 -%ster schedu&e............................................................................................................ '2.3 Resources summ%r ...................................................................................................... '2.4 echni/ues %nd methods..............................................................................................

3 *ER"F"CA"O( A#-"(" RA"*E PROCE#URE ..........................................3.1 Anom%& reportin! %nd reso&ution............................................................................ ..3.2 %s iter%tion po&ic .....................................................................................................

3.3 #evi%tion po&ic ..................................................................................................... .......3.4 Contro& procedures................................................................................................. ......3.$ t%nd%rds pr%ctices %nd conventions........................................................................ 5

4 *ER"F"CA"O( AC "*" "E ................................................................................ 104.1 r%cin!......................................................................................................................... 104.2 Form%& proofs.............................................................................................................. 104.3 Reviews........................................................................................................................ 104.4 ests.............................................................................................................................. 10

$ E REPOR "(6................................................................................................... 12

A E REA#"(E RE*"E+ ................................................................................ 14

#OCU-E( CO( RO7................................................................................................. .... 1$

#OCU-E( "6(OFF........................................................................................................ 1$

#OCU-E( C8A(6E RECOR#.................................................................................... . 1$



Review and Test Plan Page 1IDA-MS-RTPIssue 1

0 PREFACE

0.1 PURPOSE OF THIS DOCUMENT

#1 This document is a generic Review and Test Plan document for use by IDA Projects. It rovides guidance and tem late material which is intended to assist the relevantmanagement or technical staff! whether client or su lier! in roducing a

roject"s ecific Review and Test Plan document. It is also useful bac ground reading for anyone involved in develo ing or monitoring the IDA $anagement %ystem&IDA"$%'.

0.2 USE OF THIS DOCUMENT

#1 This Preface is addressed to the users of this generic document and is not meant to beretained in any roject"s ecific Review and Test Plan documents based on it.

#( The remaining sections &numbered 1! (! )!*' constitute a tem late that should beused to construct the roject"s ecific Review and Test Plan document.

Te+t in normal case is in the most art ,boiler late- that can be retained!amended or deleted in the document.

Te+t in italics rovides instructions on how to com lete a section and should beremoved once the section is written.

#) The tem late should be used ragmatically! that is " where a section is not relevant it should be omitted. onversely! the material contained in this document is notnecessarily e+haustive/ if there is a subject that is relevant to the IDA Project! but isnot included in this document! it should still be included.

#0 This document has been re ared using $% ord 23. The following variables arecurrently recorded as 4ile ,Pro erties- under $% ord. They may be modified bythat means or overwritten directly at each occurrence in the document! at thediscretion of the user.

a. “Summary” Pro ertiesTitle Ty e o! document "i.e. Review and Test Plan#Aut$or Aut$or"s# o! document%eywords Document re!erence "i.e. IDA-MS-RTP#

&. “'ustom” Pro erties

Pro( Id S$ort mnemonic o! IDA Pro(ect "set) in t$is document)to “Pro(ect Id”#Pro(ect *ull name o! IDA Pro(ect "set) in t$is document) to

“Tem late !or IDA Pro(ect”#'ontr Id S$ort identi!ier o! contract "set) in t$is document) to

“'ontract Id”#'ontract *ull name o! contract "set) in t$is document) to

“Tem late !or s eci!ic develo ment”#+ersion Issue num&er "currently Issue 1#Date Date o! document "currently 1, anuary //1#



Review and Test Plan Page IDA-MS-RTPIssue 1

0.3 OVERVIEW

#1 This reface is for information only.

#( This reface will therefore not be retained in the roject"s ecific document.

#) The remaining sections &numbered 1! (! )!*' constitute a tem late that should beused to construct the roject"s ecific document.

Te+t in normal case is in the most art ,boiler late- that can be retained!amended or deleted in the document.

Te+t in italics rovides instructions on how to com lete a section and should beremoved once the section is written.

#0 The tem late should be used ragmatically! that is " where a section is not relevant it should be omitted. onversely! the material contained in this document is notnecessarily e+haustive/ if there is a subject that is relevant to the roject! but is not

included in this document! it should still be included.

0.4 TESTING WITHIN IDA

#1 IDA rojects often vary in their com le+ity! objectives! architecture5technical solutionand outcome. They cover many subject 6 geogra hic areas and often re7uireinterfaces to be established to many e+ternal systems. In trying to create a joined"ua roach to facilitate 8uro ean decision"ma ing there are many issues to beaddressed in smoothing the movement of data 6 information across IDA networ sinto $ember %tate Administrations &$%As'! 8uro ean Institutions and other

artici ating organisations. These issues might include heterogeneous systemsenvironments and ra id technological change.

#( IDA rojects often brea new ground/ creating lin s were none were in lace before. 9in s to artici ants can sometimes be hard to s ecify and re7uire detailedclarification as the roject roceeds. This often creates situations where s ecificas ects of the system re7uirements remain undefined at the start of the roject. This isa by" roduct of the IDA rogramme and re7uires a s ecific res onse in the course ofdrafting the lanned series of tests. The re7uirements for some of these systems willevolve as the roject moves forward. are therefore has to be ta en with ee ing thetest lan under constant review to ensure that it addresses true re7uirements.

#) The testing a roach ado ted will also need to ta e into account that IDA rojectscan use differing technical solutions to rovide the interwor ing between $%As that isdesired. Traditional :fat; client a roaches! where local rocessing resides in the $%Ato draw out information that is then sent to a central server system for e+am le in

9u+embourg! may be re laced by moves towards a web"based system using :thin;clients.

#0 Test lanning has to be carried out in such a way as to ensure an a ro riate level oftesting is done to reflect the nature of the aims of the roject. A s ecific concern is theneed to address the often"wide range of interfaces that IDA systems have to interwor with in their o erational situation. The definition of how these interfaces! and thedata flows across them! will be verified in a test environment are ey elements of anytest lan.

#< In setting out to highlight the need to ta e a ositive a roach to the lanning of thetesting that will be underta en! this document see s to ensure that many of the




im ortant issues involved in ro osing a com rehensive and yet a ro riate level oftesting is carried out to assist the smooth introduction of the system under test into

service.

0.5 PURPOSE OF THE REVIEW AND TEST PLAN

#1 In order to ensure that a system will conform to its s ecifications! a rogram ofreview and testing should be lanned to cover the entire system develo ment lifecycle.

#( Throughout a roject this involves

a. tracing in ut re7uirements to their im lementation/

b. reviewing or auditing of documents to ensure they meet the standards and re7uirements assigned to them 1/

c. roving theorems used &if any'.

#) During the latter stages it also involves the staged testing in detail of art andeventually the whole of a system to rove that it meets the re7uirements laced on it.

#0 This test lan defines the a roaches! standards and rocedures that are to be used inreviewing and testing the system or networ under develo ment. In addition to

roviding a to "level lan! it addresses issues such as where test datasets should beobtained from and what can be done to simulate the o erational environment to rovethe system before it moves into an o erational situation.. This document attem ts to

set out the rinci les under which a test lan should be devised and then revised inthe course of the roject.

#< Details of actual tests to be conducted at each stage are s ecified later in the

lifecycle. These are documented in the Test % ecification document! and are notincluded in the Test Plan.

0.6 EVOLUTION OF THIS PLAN

#1 This document is re ared during the early stages of the roject to rovide reviewers!testers etc. a framewor to erform their duties. The lan should be chec ed at theend of each hase of the roject! to ensure that it is sufficiently relevant! clear! anddetailed! to allow the ne+t hase to ta e lace.

1 This does not just mean a su erficial chec that document standards have been adhered to. 4or e+am le! for a design document! there is a need to verify that the design is efficient and robustand will generate a system that meets the re7uirements laced u on it.






2 VERIFICATION OVERVIEW

#1 This section should describe the organisation 6 rocesses involved! schedule!

resources! res onsibilities! tools! techni7ues and methods necessary to erformreviews! roofs! tracing and testing.

2.1 ORGANISATION & PROCESSES INVOLVED

#1 This section should describe the organisation of the review! roofs! tracing andtesting activities for each hase. This as ect of the lanning for testing should beclear early on in the roject to ensure eo le in the roject team have time to re are

for these activities.

#( It should be made clear what rocesses should be followed to ensure the testing will

achieve the desired objectives and that the system will be fit for the ur ose for whichit was built. The relationshi s of these rocesses to the other activities such as roject management! develo ment! configuration management and 7uality assurance shouldbe described.

#) @rganisational details that should be included are=

roles and res onsibilities/

re orting channels/

levels of authority for resolving roblems.

#0 The descri tion should identify the eo le associated with the roles and highlighttheir terms of reference in the role. 8lsewhere the lan should only refer to roles.

#< $uch of the above information can sometimes be usefully summarised in table or anannotated organisation chart.

2.2 MASTER SCHEDULE

#1 This section should define the schedule for the review! roofs! tracing and testingactivities in each hase. The information may be best resented in a table or aABTT chart.

#( Bote that an im ortant milestone should be the :Test Readiness; review which shouldbe underta en rior to testing starting &see A endi+ A'.

2.3 RESOURCES SUMMARY

#1 This section should summarise the resources needed to erform reviews! roofs!tracing and testing such as staff! com uter facilities! and software tools. It may beuseful to resent this in the form of a table. A ey element here in IDA rojects will beto identify the data sources that are to be used to test the system. (

Project artici ants " $%As! agencies! etc' may re7uire time to assemble s ecific test data

and the data that will be re7uired should be notified to artici ating $%As several months in advanceof the scheduled start of the tests. The media on which the test data should be su lied will also needto be identified. Ideally a test sam le of data should be drawn off $%As systems to rove that thedata is being rovided in the right form.




2.4 TECHNI UES AND METHODS

#1 This section should identify the techni7ues and methods used for reviews! roofs!tracing and testing in each hase. This will address the e+tent to which :blac ; bo+ or

:white; bo+ testing is to be carried out at the system level.#( It should address each as ect of the system that needs to be verified! ta ing into

account its architecture and main characteristics ) . 4or e+am le! verification mightneed to cover

functionality

system installation and house ee ing

concurrency and data integrity

volumes and erformance

reliability and robustness

security.

#) Bote that it might be necessary to develo test harnesses or configure automatedtesting tools &e.g. for simulating comms traffic! or for generating load from on"lineusers'! and there will be a need to verify that these tools function correctly beforeusing them in the verification of the system itself.

0 4or e+am le! an IDA roject might re7uire a central server system to be fed by many geogra hically located :client; systems. These might be :fat; clients with a high degree of local rocessing of data held in a local database! that needs to transmit information into a central serveror to another $%A via that central server. Alternatively it might em loy a :thin; client architecture

where the client rovides a browser front end to the server and u loads data using attachments toemails etc. The a roaches to testing this system could differ 7uite significantly de ending on thearchitecture em loyed.



Review and Test Plan Page ,IDA-MS-RTPIssue 1

3 VERIFICATION ADMINISTRATIVE PROCEDURES

3.1 ANOMALY REPORTING AND RESOLUTION

#1 The rocedure for re orting and resolving anomalies found during the reviews! roofs! tracing and testing activities and the criteria for activating the anomalyre orting and resolution rocess should be defined here. The rocess by whichanomalies are ran ed in some order of riority to be addressed is crucial as the

lanning for any subse7uent testing cycle would need to cater for the wor involvedin solving any agreed list of riority faults.

3.2 TAS! ITERATION POLICY

#1 This section should define the criteria for deciding whether a tas should be re eatedwhen a change has been made. The a roach will reflect the assessment of thecharacteristics of the system as it will differ de ending u on the nature of the solutionthat has been devised/ for e+am le! a tightly"cou led system might re7uire a differenta roach to a loosely"cou led system.

#( The criteria may include assessments of the sco e of a change! the criticality of the function&s' affected! and any 7uality effects. In testing terms this is very im ortant! asan assessment needs to be made of which elements of the system may be im acted bya change. The Tas Iteration Policy would have to address how to roceed in theevent that there are areas which would be clearly unaffected by a change. It is

ossible that either these can be missed out in any subse7uent testing cycle or that a small number of tests that are considered re resentative of those needed to establishthat the system o eration has not fundamentally changed can be identified.

#) This section may also need to refer to the develo ment contract or s ecificationagreed with the develo er which might include agreed criteria for re"tests.

3.3 DEVIATION POLICY

#1 This section should describe the olicy for deviating from the lan! and define thelevels of authorisation re7uired for the a roval of deviations. The informationre7uired for deviations should include tas identification! deviation rationale and the

effect on software 7uality.

3.4 CONTROL PROCEDURES

#1 This section should identify the configuration management controls for the out uts ofreview! roofs! tracing and testing. Ade7uate assurance that they are secure fromaccidental or deliberate alteration is re7uired. An im ortant facet of this will be thearchive olicy for the test results from both the initial hase of testing and any

subse7uent testing that is carried out as the system moves on into service and ismaintained.




3.5 STANDARDS" PRACTICES AND CONVENTIONS

#1 This section should identify the standards! ractices and conventions that governreview! roof! tracing and testing tas s! including internal organisational standards!

ractices and olicies.




4 VERIFICATION ACTIVITIES

#1 This section should describe the rocedures for review! roof! tracing! and testing

activities.

4.1 TRACING

#1 This section should describe the rocess for tracing each art of the in uts to theout uts! and vice"versa i.e.=

4orwards traceability " which re7uires that each in ut to a hase shall betraceable to an out ut of that hase! and should demonstrate com leteness.

Cac wards traceability ? which re7uires that each out ut of a hase shall betraceable to an in ut to that hase.

4.2 FORMAL PROOFS

#1 This section should define or reference the methods and rocedures used &if any' for roving theorems about the behaviour of the software. The a roach here will beheavily influenced by the characterisation of the system outlined at the early stage ofthe roject when the Architecture of the system has been established.

#( ey as ects of the formal roofs will be to loo at how stable the system is and how itwill react under different failure modes. A line failure or data failure should notcreate an unstable system. 87ually formal roofs may also have to address security

issues where ossible : enetration testing; might need to be carried out to ensure the systems are not vulnerable to attac from malicious users.

4.3 REVIEWS

#1 This section should define or reference the methods and rocedures used for technical reviews! wal throughs! software ins ections and audits. It may be a ro riate to refer to the IDA"$% uidance on Reviews &IDA"$%"R8E'.

#( A list of the reviews! wal throughs and audits that will ta e lace during each haseand identify the roles of the eo le artici ating in them! should also be identifiedhere.

4.4 TESTS

#1 This section should define or reference the methods and rocedures used for testing.

#( The section should define the level of testing to be carried out for each hase. Thelevel of testing that might be e+ ected is shown below=

Unit Testing " 4ull Design Re7uirements overage &Clac Co+'! and in caseswhere code is critical full statement coverage as well.

Integration Testing " 8+ercising all interfaces! and all arameters

System Testing " overage of all re7uirements from the %ystem Re7uirement

Acceptance " overage of all re7uirements from the Fser Re7uirement Document



Review and Test Plan Page 1/IDA-MS-RTPIssue 1

5 TEST REPORTING

#1 This section should describe how the results of im lementing the lan will be

documented. This will de end u on the %oftware Develo ment $odel selected for the roject.

#( This as ect of the wor should recognise that testing is rarely a singular event andthat this will be true for IDA related develo ments. %ystems often go through cycles of testing and whilst a set of re orts may be re7uired for the first cycle of testing theymay not all be re7uired in future cycles. It may be that after a number of cycles oftesting! over a eriod of several years! a system may only re7uire a single re ort! suchas the Test re ort! to be generated. Ty es of re orts that might be generated for aclassical waterfall based roject include=

summary re ort for the hase/

technical review re ort/

wal through re ort/

audit re ort.

test re ort

#1 The Summary Report for the hase may be between five and ten ages in length. Itwould cover the major oints drawn out from the testing. It is suggested that there ort be structured with a single highlight sheet at the front based on a smallcommentary and a table that will show ey areas where successful testing has beencom leted and areas where concern must be documented. The re ort should alsoaddress any issues that arose from e+ternal de endencies that may have affected the

schedule. In IDA rojects there may be s ecific factors that need to be highlighted such as delays to obtaining data to be used for the testing etc.

#( A Technical Review Report would focus in greater detail on any technical issues thathad arisen from the testing. This might address erformance issues and any li elyindicators from the testing as to the nature of the roblems! bottlenec s etc.! and

ossible solutions.

#) The Walkthrough Report would contain the results of a visual ins ection of the codeinvolved in the roject. This may be carried out randomly in the code to rovidechec s of the 7uality of documentation and to ensure that articular failure modesthat might arise from the normal use of the system! such as a communications line

fault! can be handled in an orderly way by the software.

#0 The Audit Report would rovide a view from the stand oint of the entire systemdocumentation su orting the introduction of the system into service. It would assess

from an inde endent view oint to what e+tent the testing was li ely to cover all of the ossible o erating modes that the software may enter.

#< The Test Report could contain a review of the outcomes of the testing. It couldcontain statistical evidence of articular failures! assessing them from the view ointof their severity and im act u on o erational erformance. 4ailures could beassigned to a category that indicates the need to resolve the roblem highlighted inthe testing and the urgency with which it should be addressed/ for e+am le! a systemrating failure modes from a grade A fault! which means that the system is inca able




of o erating effectively! through to a grade 8 fault! which is assessed has having noim act u on the o erational use of the system. It is suggested that this is the onere ort that is always generated as a result of entering another cycle of testing.

#G In contrast if it is felt a ro riate to ado t a RAD a roach to the softwaredevelo ment rogramme then other documents will be more a ro riate. RAD isli ely to be used on less com le+ systems where one is initially develo ing a ilot

system rior to going into an o erational environment.

#3 In this case re orts might be generated at the end of each :Timebo+; as a summary ofthe failures detected and the nature of the im act on the o erational system and anyre7uirements that emerge to be included in the ne+t :timebo+;. At the end of the

roject a standard set of re orts! such as those ro osed for the aterfall a roach! should be generated ta ing into account any intermediate results ta en as sna shotsat the end of each timebo+.

#H If the :E; %oftware Develo ment $odel is used then it will be im ortant to reflect thenature of the verification that needs to ta e lace at each stage of the develo mentcycle. Anomaly re orts should be attached to the a ro riate verification re ort ateach stage of the rocess.




A TEST READINESS REVIEWS

#1 At this review the results of various other stages of review! such as code wal throughs!

module 6 subsystem testing should be considered and the log of the rocessesunderta en to lace the software under configuration control.

#( It is im ortant that such a review is held as often IDA rogrammes are 7uite com le+and re7uire many $%As to be drawn together to develo the schedule and be involved in ensuring that their art of the overall system! such as a s ecific communicationslin ! is installed.

#) It is suggested that a chec list be drawn u of the items that should feature in the :test readiness; review. These would include=

Architecture characteristics assessment

Results of module tests Results of %ubsystem tests

Results of any testing on the test harness ? if re7uired

Details of the rocesses used in lacing the software under configuration control

Results of the analysis of faults that have been fi+ed since the last cycle of systemtesting.

#0 The actual rocess followed will have to reflect the a roach being ta en to thedevelo ment of the system! the chosen %oftware Develo ment $odel! and thecharacteristics of the architecture of the solution.

#< In the case where the :waterfall; a roach is being used this rocess will need to beunderta en a small number of times as the system cycles through testing! faults beingidentified! those faults being re aired and then re"submitting the system for furthertesting. In the case where RAD is used this rocess will be more su erficial as each:timebo+; comes to a close.

#G Reviews of large systems should be bro en down by subsystems. During the design hase! critical design reviews of subsystems should be held when the subsystem isready! not when all subsystems have been designed.




DOCUMENT CONTROL

it&e9 Review and Test Plan

"ssue9 Issue 1

#%te9 1, anuary //1

Author9 *rances Stevens) 'olin Todd) Dave Sloggett

#istri)ution9 6' D7 6nter rise 8 7avino MurgiaPro(ect Team

Reference9 IDA-MS-RTP

Fi&en%me9 /315500.doc

Contro&9 Reissue as com lete document only

DOCUMENT SIGNOFF(%ture of i!noff Person i!n%ture #%te Ro&e

Aut$ors *rances Stevens)'olin Todd) DaveSloggett

Pro(ect Mem&er

Reviewers o$n 9rin:wort$ Pro(ect 'ontroller

DOCUMENT CHANGE RECORD#%te *ersion Author Ch%n!e #et%i&s

/3 Decem&er /// Issue 1 Dra!t 0 Dave Sloggett Internal Review

Decem&er /// Issue 1 Dra!t Mar: Pillatt Review and tidy u

1/ anuary //1 Issue 1 Dra!t 2 Sue Turner ; dated !ormatting

1, anuary //1 Issue 1 Mar: Pillatt Issue

Documents

Review and Test Plan