SOLAR ORBITER SOC Test Plans Nana Bach SOWG#7 – 6-9 July 2015

SOLAR ORBITER SOC Test Plans

Nana Bach

SOWG#7 – 6-9 July 2015

Contents

• Solar Orbiter SOC Test Plan

• Identification of different Test Levels

• SOC Internal Testing Process

• Continuous Delivery approach

• Requirements Tracing

• SOC Interface Testing

• Interaction with different groups

• System Test

• Conclusions

SOWG#7 – ESAC – 6-9 July 2015

Solar Orbiter SOC Test Plan

• Describes the strategy, scope, approach, resources and schedule of the testing activities within the SOC.

• Identifies all the needed levels of testing, processes, actors and harnesses.

• Identifies the deadlines for readiness of the SOC.

• Identifies the roadmap to fulfil the System Requirements and calculate the traceability.

• Describes the testing roles and responsibilities, along with the technical activities, interfaces, and scheduling.

• Is based on SVVP (System Verification and Validation Plan) which identifies the Verification and Validation Strategy and classifies the SRD requirements and their methods of Verification and Validation.

• Will trace all the requirements identified in SVVP as to be verified by “Testing”.


Solar Orbiter SOC Test Plan


SVVP

Software Product Test Specification

Template

Software Test Specification 1

Software Test Specification 2

Sub-System Test Specification

Template

Sub-System 1 Test Specification

Sub-System 2 Test Specification

External Interfaces Test Specification

Template

Interface to MOC Test Specification

Interface to Instrument Teams Test Specification

System Test Specification

Template

Operational Rehearsal 1 Test Specification

End-to-end Test Specification

SOC Test Plan

Software Product Test Report Template

Softwar e Test Report 2

Software Test Report 1

Sub-System Test Report Template

Sub-System 1 Test Report

Sub-System 2 Test Report

External Interfaces Test

Report Template

Interface to Instrument Teams Test Report

Interface to MOC Test Report

System Test Report Template

Operational Rehearsal 1 Test

Report

End-to-end Test Report

• Document Tree:

Identification of different Test Levels


System

Sub-System 1

Software Product 1

Software Product 2

Sub-System 2

Software Product 3

External Interface

Software Product 4

• The Solar Orbiter SOC is identified as a System composed by different Sub-Systems.

• Each Sub-System can have one or multiple Software Products beneath.

• Smallest identified components to be tested are the Software Products. • The functionalities, processes and different Software Products are grouped by the Sub-Systems into one functional/procedural unit.

• The Sub-Systems which require involvement of External Parties are called External Interfaces.

• We distinguish between externally and internally developed SP.


Therefore the 4 identified levels of Testing are:

1. Software Product Level Test

Unit Tests

Functional Tests

Non-Functional Tests

• Performance

• Usability Tests

• Reliability Tests

• Robustness Tests

• Load Tests

• Stress Tests

Integration Tests

Regression Tests



1. z

2. Sub-System Level Test

Use Case Testing

Sub-System Validation

Sub-System Integration

3. External Interfaces Level test

MOC-SOC Interface Tests

Instrument Teams Interface Tests

Science Archive Interface Tests

4. System Level Test

End-to-End Tests

Operational Rehearsals

SOVTs / Post-SVTs


SOC Internal Testing Process


SystemSRD

Sub-System 1Use Cases

Software Product 1

User Stories

Software Product 2

User Stories

Sub-System 2Use Cases

Software Product 3

User Stories

External InterfaceUse Cases

Software Product 4

User Stories

• Each logical group of functionalities have been described in Use Cases. • The Sub-Systems Test Level can be split into the Sub-System Use Case testing as well as the Sub-System Validation. • Use Cases can be integrated into a bigger functional pieces, building up to full Sub-System. • Tests against the external interfaces to the SOC like e.g. The Instrument Teams or MOC are External Interfaces Tests.• System Level Tests involve the whole System e.g. the SOVTs/SVTs, Operational Rehearsals, End-to-End tests etc.• The SRD is linked to the Use Cases.• The Use Cases are split into User Stories and implemented within SCRUM development technology.

SOC Internal Testing Process

Steps of the development/testing process:

1. In Sprint Planning Meeting decompose Use Cases into functional blocks.

2. Implement the code corresponding to each functional block.

3. Implement the acceptance test (Cucumber) corresponding to the peace of code.

4. Run the test in Hudson/Jenkins as part of automated build.

5. When the entire Use Case is ready perform the test on Sub-System Level.

6. Validate the Sub-System when all the Use Cases implemented.

7. Integrate Sub-Systems into the System (or parts of it).

8. Perform the External Interfaces tests.

9. Perform the System Test.


Continuous Delivery Approach


• Continuous Integration and Delivery approach.

• Automated build, test and deployment approach.

• Behaviour Driven Development strategy (BDD).

• Continuous integration and test framework Hudson and Jenkins.

• Test integration and functional framework Cucumber.

• Feature implementation leads to automated testing.

• Releases of Software Products on automated basis.

• The Sub-System testing automated as much as possible.

• Automated Use Case testing whenever possible.

SW Develop

ment

Unit Test

Suite of FeatureAccepta

nce Tests

Release/Deploym

ent

Requirements Tracing


• SIRD Requirements are linked to the System Requirements.

• The System Requirements are summarized after their scope.

• No breaking down by the Sub-Systems.

• Test Specification Documents will contain traceability to the SRD.

• Each of the Test Cases will link to one or multiple System Requirements.

• Test Reports will show which of the Requirements have been tested.

• The System Requirements will be classified into different categories.

• Requirement Verification and Validation Matrix in the SVVP.

Requirements Tracing


• No further breakdown on the System Requirements.

• Software Requirements for each of the Software Products summarized in the Cucumber Features.

• The Use Cases are going to be decomposed into different User Stories (within the Jira, SCRUM methodology) and summarize one single functionality or group of functionalities, which can be considered as “functional” Software Requirements.

• Described in the Feature Description within the

automated Test framework Cucumber.

• For non-functional Software Requirements

a Cucumber feature will also exist.

• The summary of all the Cucumber features

for a given Software Product will represent

its Software Requirements.

SOC Interface Testing

• Identified Interfaces:

• MOC (SOC External Interface)

• GFTS

• EDDS

• Instrument Teams (SOC External Interface)

• GFTS Server

• Low Latency Pipelines (also considered as externally delivered Software Product)

• Low Latency Pipelines Visualizations

• Science Archive (SOC Internal Interface)


Interaction with different groups

• Different groups:

• SOC

• MOC

• Instrument Teams/PIs

• External developers/possible COTS

• Define the Interfaces (ICDs)

• Assure the Interfaces have been implemented and tested correctly (External Interfaces Test Specifications and corresponding Test Reports).

• Define the communication channels between the different groups.

• Define Operational Procedures.

• Validate Operational Procedures.

• Define Procedures to be followed in cases of anomalies/disaster.

• Identify clear milestones in the schedule.


System Test

1. End-to-End tests

• Test the System as a hole

• Use when possible realistic (or even real) data sets

2. Operational Rehearsals

• Test Operational Procedures

• Define Operational Scenarios/Environments

3. SOVTs/ Post-SVTs

• Verify the readiness of the SGS with the SOVTs

• Use the output of the SVTs run by the MOC

=> Formal TRR/TRB procedures to be held for each of the System Tests


Conclusions

A lot of work still to be done …
