Britta Roll, DLR Braunschweigtau.dlr.de/fileadmin/documents/meetings/2008/pdf/5... · A test matrix for the FlowSimulator using TAU-Python (to be defined) A test matrix on the Airbus

1

Slide 1 > TAU-QA > B.RollQuality.ppt >12.02.2008

TAU quality assurance

Britta Roll, DLR BraunschweigInstitute of Aerodynamics and Flow Technology


TAU quality assurance

Introduction, overview

TAU software integration process

TAU test matrices

Error handling

New or modified TAU input parameters

Documents

2





What is Software Quality Assurance?

There are different descriptions to define Software Quality Assurance.

The correct definition goes something like:

“The function of software quality that assures that the standards, processes, and procedures are appropriate for the project and are correctly implemented.”

This definition is taken from Software Definitions at NASA.

3



Another example for a definition:“Software Quality Assurance consists of a means of monitoring thesoftware engineering processes and methods used to ensure quality.“

Further example for a definition:“Software Quality Assurance includes the process of assuring thatstandards and procedures are established and are followed throughout the software acquisition life cycle.”

Some benefits of Software Quality Assurance:Improved customer satisfactionReduced cost of developmentReduced cost of maintenance



What is Software Quality?

„Quality software is reasonably bug-free, delivered on time and within budget, meets requirements and/or expectations, and is maintainable.”

(This definition is taken from AskNumbers.com .)

Objective: The TAU-Code should be quality software.

4



Some standards and procedures which has been established for thedevelopment of the TAU-Code to assure its quality:

Quality Management of the TAU-Code developmentdescribes the procedures to be applied to the development, the responsibilities, communication, accomplishment of the development, configuration management and the applying documents

Configuration management procedure (SVN-repository)Release management procedure (code versioning)Internal software verification&validation concept (TAU test matrices)Bug management procedure (Mantis)



List of documents for the development of the TAU-Code to assure its quality:Airbus/DLR support conceptQA Documents

QM plan of the TAU-Code developmentTAU software integration standardSoftware development planProgramming guidelinesBug managementTAU-Code change request procedureTests results after releaseTAU User GuideTAU Technical DocumentationChanges document (RELEASE)

5



The following slides describes some quality assurance procedures which should be applied by TAU developers: the TAU software integration process (and the programming guidelines), the software testing (using the TAU test matrices), the bug management, and the management of TAU input parameters.



6



Overview:

The document ‘TauSoftwareIntegration.doc’ describes the software integration process for the TAU-Code.

This process defines the procedure and the related rules for theintegration of new or modified source code into the actual SVN-repository.

The defined procedure is to be applied to the development of the TAU-Code for assuring its quality.

The actual SVN-repository contains the central TAU version.



Overview:

Input for the integration process are:New software developmentsSoftware modifications

A software development plan for the TAU-Code, for which a steering committee is responsible, lays out which software developments and/or modifications are allowed.

The software development is distributed over different developer sites.All sites use a common procedure for the software integration. At each site a local coordinator is responsible for the local software integration process and the harmonization with software integration on other sites together with the other site coordinators.

7



Local coordinators:

The local coordinators support the steering committee in defining the software development plan. They also manage the harmonization of work and the work share between the sites in a pre-release testing phase.Local coordinators: Ralf Heinrich (Braunschweig),

Thomas Gerhold (Göttingen).

The development of the TAU-Code is coordinated within developer meetings, chaired by one local coordinator and taking place regularly.Within the developer meetings the software development plan is updated.



Local software integration process:

The procedure for the local software integration process is described as a list of check points, divided into three integration stages.The developer and the local coordinator are both responsible forassuring that the integration process is according to this procedure.

Note: It is useful that the developer at first propose a concept about his development work, which he/she should harmonize with the local coordinator.

8





List of check points for the first stage of the integration:

new software development or modified implementationeach change should lead to a better or equal code than before either due to new functionality or due to improved implementation:

better code means more efficient, better structure or more generality, never less functionality.

no memory and no runtime overhead for existing functionalityverification of algorithms on a suitably selected development test case

cont.

9




no unnecessary change in usage, change in keyword names for input parameters, etc.no unnecessary renaming of source files in order to keep SVN-history as much as possiblecoding as modular as possible, use of lib-functions, no code-doubling, etc …coding in accordance with programming rulescoding in accordance with existing code-philosophy – one example is that the implementation allows for parallel mode (or at least for parallelization in case of stepwise integration)

cont.



What are the programming rules?

The programming guidelines comprise the programming rules for coding, commenting and naming and assist with the maintenance and extension of the TAU source code.These guidelines are the common standard for the programming style which should be used for the development of TAU for assuring itsquality. These guidelines can be found in the SVN-repository https://tausvn.as.dlr.de/repos/dlr/tau/trunk/ptau3d_doc/TEXT/- in German: ProgrammierRichtlinien.tex- in English: ProgrammierRichtlinien_english.texThey can be found as pdf-documents at the TAU-Site (Webpage).

10




code compiles without warnings by use of pedantic warning level, check if possible different compilersruntime relevant parts of the code are checked with respect to parallelization or to cache-optimization on scalar architecturestest matrix runs without unintended side effects and without change of results

in case of algorithmic changes which change results compared to the actual version a validation on several selected test cases has to proof correctness

major code changes to be tested for different architectures and different compilers

cont.




in case of severe changes with impact on many source files or different code-modules the development should be announced to all developers, e.g. in a developer meeting or via the local coordinators

if code is changed in which not all details of the previous implementation are very clear, clarification by discussion in the developer team or via local coordinator is required

11



List of check points for the second stage of the integration:

get clearance for check-in from local coordinator by checking the changes versus “first stage check points”



List of check points for the third stage of the integration:

update code to (merge with) actual SVN-versioncheck for test matrix run without unintended side effects for updated implementationSVN-check-in with meaningful log-message for indicating all aspects with respect to changes of results for release-change file, changes in usage for release parameter-change fileimmediately after SVN-check-in a complete check-out and compilation of all is to be performed to ensure that no new files or SVN-modules or changes have been forgotten during check-in.

cont.

12



List of check points for the third stage of the integration:

update of test matrix reference resultsupdate of User Guideupdate of Technical Documentationcheck next-morning results of automatic test matrix for errors in integration - in case of errors or compiler warnings update of check-in with highest priority. The local coordinator has to ensure that the check-in update is done.


TAU test matrices

13


TAU test matrices

For quality assurance of the TAU-Code there exist several test matrices:The test matrix (called TauTest) for the development of TAUThe Airbus test matrixA test matrix for TAU-PythonA test matrix for the FlowSimulator using TAU-Python (to be defined)A test matrix on the Airbus site

Each test matrix consists of several test cases to verify the capabilities of the TAU-Code. The TauTest matrix and the Airbus test matrix cover a representative field of applications from DLR and Airbus site. Every TAU developer should use the TauTest matrix before check-in.


Features of the TauTest matrix

A test matrix, which runs every night automatically using the TAU-Code in the actual SVN-repositoryThe test results of a run are compared with reference resultsA test summary of the differences is sent to all TAU developersSequential and parallel test casesTest matrix consists of two levels of complexity:

Test case level 1, postfix -L1: very fast execution only a few iterations from scratch or restartTest case level 2, postfix -L2: fully converged 2D computations from scratch or a few iterations for a larger 3D case

14


Using the TauTest matrix

The SVN-repository for TAU includes this test matrix inclusive the reference results (tar file ‘References.tar’) and a README for using the test matrix

https://tausvn.as.dlr.de/repos/dlr/tau/trunk/TauTest

Each developer should check-out this test matrix and use it locally for his development work

If new functionality will be implemented in TAU, the developer should extended the test matrix by adding a suitable test case (see theinstructions in NEW_TEST_CASE_HOWTO).


Using the TauTest matrix locally

The developer has to check the results of the local test matrix run with the changed TAU-Code, in order to avoid unexpected errors or unintended side effects.

No (significant) differences should be found for test cases of level 1.Reference version: TAU-Code in the actual SVN-repositoryTest version: changed TAU-Code (including the development work)

Note: It is possible that (additional) differences between the reference results and the test results appear, which are caused by different floating-point arithmetic's, if the test results are produced on another computer than the reference results.

15



Procedure in short form for using the TauTest matrix locally:Check-out the test matrix from the actual SVN-repositoryRun the test matrix once with the unchanged TAU-Code: make testSet the current test data as new local reference data: make setrefModify the TAU-Code according to your development workTest the updated implementation by running the test matrix with the changed TAU-Code and by comparing with the local reference data: make checkIf test results are ok, check-in the modified TAU-Code (see third stage of the TAU integration process).



Continuing the procedure:Check next-morning results of the test matrix which runs every night – for differences and particularly for errors in integration.In case of errors or compiler warnings update of code check-in with highest priority!If there are no errors and occurring differences between the reference results and the test results are acceptable (e.g. caused by codecorrection), a new tar file with the new complete reference data has to be provided to all:1) login to the machine on which the test matrix is running automatically,

the directory /home/tau/tau/tarref contains all reference tar files2) check-in the tar file from the last test matrix run as ‘References.tar’ in

the actual SVN-repository

16


Error handling


TAU error handling with Mantis

For reporting functional errors of the TAU-Code and problems which should be corrected there exists the DLR bug-fixing forum Mantis.

Mantis is a web-based bug tracking system that utilizes a database with an easy to use web interface. All bugs, which are reported by a developer or a user, are automatically saved in this database. Mantis ensures that each bug is tracked to closure and verified.

Mantis Webpage: http://taubt.as.dlr.deDocumentation (only in German) in a directory of the TAU-Code:taudir/doc/mantis-bugtracking-german.pdf

17


TAU error handling with Mantis



18



There exist a parameter database for the TAU-Code in BibTEX format.

This database contains all parameters that are valid in the code and is handled by a python-script-set named TauPDB.This database simplifies the update of parameter documentation to make the User Guide more up-to-date and useful. TauPDB also offers several functions to query the database. It provides the ability to query the detailed information as described in the User Guide online.

If a developer wants to add a new parameter or modify an existing parameter in the TAU-Code he/she should maintain this parameter database.



The documentation about the parameters and TauPDB can be found in the User Guide:

chapter 3 Parameter filechapter 4 Input parameter overviewchapter 14 TAU-Code parameterfile database

The python-script-set TAUPDB and the corresponding README_pdb can be found in directory taudir/py_utils of the TAU-Code. The first part of this README describes how to maintain the database.

The directory taudir/input/para_dbs contains the actual database.

The sample parameter files in directory taudir/input might also be modified by the developer.

19



A few examples for modifications of the parameter database:add a new parameter (new keyword and new value)delete an old parameter modify an existing parameter by:

changing the keywordchanging the valuechanging the scope of the valuesadding a new value or deleting an existing value (this requires also a change of the scope)setting a new default value deleting the previous setting of the default valuechanging the type of the parameter


Documents

20


Documents

The quality assurance documents which are useful for TAU developer can be found at the TAU-Site (Webpage).Additionally the documents can be found in the directory /home_as/nv/taubs/TAU_DIR/QA (only located in Braunschweig)

List of documents:ProgrammierRichtlinien.pdf (in German)ProgrammierRichtlinien_english.pdf (in English)TauSoftwareIntegration.doc (in English)QS_CodeChangeRequestProcedure.pdf (in English)

The last document describes the procedure for code change requests which are be applied for improvements of the code format.