Test generation from functional 3D virtual environment models · 2014-09-18 · Test generation...

Test generation from functional 3D

virtual environment models

Barath Kumar1, Rainer Drath2 and Juergen Jasperneite3

(Contact: bkumar@dspace.de)

1 inIT – Institute of Industrial Information Technologies, Germany (former employer).

1 dSPACE GmbH, Paderborn, Germany (current employer).

2 ABB Corporate Research Center, Ladenburg, Germany.

3 Fraunhofer IOSB – INA, Anwendungszentrum Industrial Automation, Lemgo, Germany.

2 | 17.09.2014 | Barath Kumar – Test generation from functional 3D virtual environment models

Objective of the prototype

Automated test case generation from 3D virtual production floor models (plant models) for Programmable logic Controller (PLC) testing.

3 | 17.09.2014 | Barath Kumar – Test generation from functional 3D virtual environment models

To Test

Control software of:

Migrating PLCs (i.e. PLCs with updated logic)

Exchanged PLCs (i.e. New hardware without changes to logic)

4 | 17.09.2014 | Barath Kumar – Test generation from functional 3D virtual environment models

Errors to be identified in the control logic:

• Logical Errors

• e.g. If the tray is empty, does the PLC take an appropriate action?

• Timing Errors (Not considered in this work)

• e.g. If the tray is empty, does the PLC take appropriate action in a pre-defined duration?

• Plant Errors

• e.g. if a sensor is broken can the PLC still handle the plant appropriately

5 | 17.09.2014 | Barath Kumar – Test generation from functional 3D virtual environment models

Haves and Challenge

• A 3D virtual representation of a production facility (plant model). This virtual facility can mimic the workings of the real plant – when we provide the necessary stimulus

• We know how to generated test cases from formal behavior models

• To extract the necessary behavior information from the 3D virtual plant model to construct formal behavioral model of the plant

Our Challenge:

Haves:

6 | 17.09.2014 | Barath Kumar – Test generation from functional 3D virtual environment models

Proposed Approach

Plant Model (Robot Studio Model)

Formal Model

of the Plant

Behavior

(Test Model)

Learning

Test Case

Generation

Test Suite

Test Case

Execution

PLC

7 | 17.09.2014 | Barath Kumar – Test generation from functional 3D virtual environment models

Proposed Approach

Plant Model (Robot Studio Model)

Formal Model

of the Plant

Behavior

(Test Model)

Learning

Test Case

Generation

Test Suite

Test Case

Execution

PLC

8 | 17.09.2014 | Barath Kumar – Test generation from functional 3D virtual environment models

Hardware-in-the-loop (HIL) Simulation - Information Source

Plant Model

Reference PLC

Industrial

Bus

Hardware-in-the-Loop Model Learning Setup

9 | 17.09.2014 | Barath Kumar – Test generation from functional 3D virtual environment models

Model Learning

Learning

Signal Analyser

Plant Model (Robot Studio Model)

Formal Model

of the Plant

Behavior

(Test Model)

10 | 17.09.2014 | Barath Kumar – Test generation from functional 3D virtual environment models

Virtual Plant Model

Sensor 3

Sensor 2

Sensor 12

Sensor 4

Sensor 11

11 | 17.09.2014 | Barath Kumar – Test generation from functional 3D virtual environment models

Information from Signal Analysis

Sensors (State Variables)

12 | 17.09.2014 | Barath Kumar – Test generation from functional 3D virtual environment models

Information from Signal Analysis

State Variables

and so on..

Event Occurrence

e.g. State Structure: 0 0 0 0 0

s2

s3

s12

s4

s1100000 00010

00011

13 | 17.09.2014 | Barath Kumar – Test generation from functional 3D virtual environment models

Formal Plant Model (Test Model)

00000

0001000011

e.g. Structure: 0 0 0 0 0

s2

s3 s4

s11s12

14 | 17.09.2014 | Barath Kumar – Test generation from functional 3D virtual environment models

Plant Model (Robot Studio Model)

Formal Model

of the Plant

Behavior

(Test Model)

Learning

Test Case

Generation

Test Suite

Test Case

Execution

PLC

Test Case Generation

15 | 17.09.2014 | Barath Kumar – Test generation from functional 3D virtual environment models

Test Case Generation (with coverage criterion)

Test case

generationCoverage

criterion

Q

Test case

is the

Coverage criterion

met?

No

Yes

1

3

2

4

5

a

b

c

def

g

h

Test model/Automaton Final_test_suite

{

// All paths from state 1 to

state 5

a ; b ; h ;

a ; b ; e ; f ;

a ; d ; f ;

}

Test suite

16 | 17.09.2014 | Barath Kumar – Test generation from functional 3D virtual environment models

Generated Test Suite

Module Test Suite

s3 = 1;s2 = 1;s3 = 0;s12 = 1;s2 = 0;s12 = 0;

s3 = 1;s2 = 1;s3 = 0;s2 = 0;

s3 = 1;s3 = 0;

s2 = 1;s12 = 1;s2 = 0;s12 = 0;

s2 = 1;s3 = 1;s3 = 0;s12 = 1;s2 = 0;s12 = 0;

s2 = 1;s3 = 1;s3 = 0;s2 = 0;

s2 = 1;s2 = 0;

s11 = 1;s11 = 0;

s11 = 1;s4 = 1;s4 = 0;s11 = 0;

s11 = 1;s4 = 1;s11 = 0;s11 = 1;s4 = 0;s11 = 0;

s11 = 1;s4 = 1;s11 = 0;s4 = 0;

s4 = 1;s11 = 1;s4 = 0;s11 = 0;

s4 = 1;s11 = 1;s4 = 0;s4 = 1;s11 = 0;s4 = 0;

s4 = 1;s11 = 1;s11 = 0;s4 = 0;

s4 = 1;s4 = 0;

Test case generation time: 00:00:45.5781250 seconds

Avoided loops : 8

Generated test cases: 15

17 | 17.09.2014 | Barath Kumar – Test generation from functional 3D virtual environment models

Plant Model (Robot Studio Model)

Formal Model

of the Plant

Behavior

(Test Model)

Learning

Test Case

Generation

Test Suite

Test Case

Execution

PLC

Test Case Execution

Completes the process

but not part of this

pilot project

18 | 17.09.2014 | Barath Kumar – Test generation from functional 3D virtual environment models

Analysis of the Approach

• Errors in ‘Migrating PLCs’ and on ‘Exchanged PLCs’ can be detected

• No manual formal modeling effort is required

• This approach can be adopted even for testing legacy systems thathave been in existence for decades

• Existence of a reference implementation (‘Reference PLC’) is a pre-requisite

• As the reference implementation is the input for the learned model,logical errors present in the reference implementation cannot bedetected

19 | 17.09.2014 | Barath Kumar – Test generation from functional 3D virtual environment models

Outlook

• Learning of continous behavior

• Learning of non-deterministic behavior

• Learning of real-time behavior

• Test case generation from complex ‘non-deterministic hybrid real-time’ models

• Identification of timing errors

20 | 17.09.2014 | Barath Kumar – Test generation from functional 3D virtual environment models

I am done !

21 | 17.09.2014 | Barath Kumar – Test generation from functional 3D virtual environment models

Reserve Slides

22 | 17.09.2014 | Barath Kumar – Test generation from functional 3D virtual environment models

Information from Signal Analysis

State Variables

and so on..

Event Occurrence

e.g. State Structure: 0 0 0 0 0

s2

s3

s12

s4

s1100000 00010

00011

23 | 17.09.2014 | Barath Kumar – Test generation from functional 3D virtual environment models

Formal Plant Model (Test Model)

00000

0001000011

00001

01000

10000 11000

10100

00100

e.g. Structure: 0 0 0 0 0

s2

s3 s4

s11s12

24 | 17.09.2014 | Barath Kumar – Test generation from functional 3D virtual environment models

Benefits

• Cost-effective

• Reduces plant down-time (planned and unplanned downtime)

• Surprises during commissioning can be minimized

• To improve testing thoroughness and maintenance of tests cases

25 | 17.09.2014 | Barath Kumar – Test generation from functional 3D virtual environment models

Analysis Summary

Logical Errors Plant Errors

Reference PLC No Yes

Exchanged PLC Yes Yes

Migrating PLC Yes Yes

26 | 17.09.2014 | Barath Kumar – Test generation from functional 3D virtual environment models

Test Case Generation (Basic Idea)

1

3

2

4

5

a

b

c

def

g

hTest case

generation

Test Model / Automaton

Test_case

{

a ; b ; h ; g

}