24
© author(s) of these slides 2008 including research results of the research network ES and TU Darmstadt otherwise as specified at the respective slide 08.05.2014 Johannes Bürdek [email protected] Tel.+49 6151 16 76089 ES Real-Time Systems Lab Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of Computer Science (adjunct Professor) www.es.tu-darmstadt.de Model-based Test Case Generation for (Dynamic) Software Product Lines The 6th Meeting on Feature-oriented Software Development 2014, May 04-07 – Schloss Dagstuhl, Germany DFG Priority Programme 1593 Design for Future – Managed Software Evolution

Model-based Test Case Generation for (Dynamic) Software … · 2014-05-08 · Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of

  • Upload
    others

  • View
    6

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Model-based Test Case Generation for (Dynamic) Software … · 2014-05-08 · Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of

© author(s) of these slides 2008 including research results of the research network ES and TU Darmstadt otherwise as specified at the respective slide

08.05.2014

Johannes Bürdek

[email protected]

Tel.+49 6151 16 76089

ES Real-Time Systems Lab

Prof. Dr. rer. nat. Andy Schürr

Dept. of Electrical Engineering and Information Technology

Dept. of Computer Science (adjunct Professor)

www.es.tu-darmstadt.de

Model-based Test Case Generation for

(Dynamic) Software Product Lines

The 6th Meeting on Feature-oriented Software Development 2014, May 04-07 – Schloss Dagstuhl, Germany

DFG Priority Programme 1593 Design for Future – Managed Software Evolution

Page 2: Model-based Test Case Generation for (Dynamic) Software … · 2014-05-08 · Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of

ES – Real-Time Systems Lab 2 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines

Testing Software Product Lines

P1 = {f1} P1 = {f1} P2 = {f2} P2 = {f2} P3 = {f1,f3} P3 = {f1,f3}

S1 S2

S4

S1 S2

S4 S1 S2

S3 S5

S1 S2

S3 S5

… …

Pro

ducts

P

roducts

Test

Models

Test

Models

… …

Test

Suites

Test

Suites

S1

S3

S2 S1

S3

S2

[Cichos et al. 2011]

Page 3: Model-based Test Case Generation for (Dynamic) Software … · 2014-05-08 · Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of

ES – Real-Time Systems Lab 3 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines

Testing Software Product Lines

P1 = {f1} P1 = {f1} P2 = {f2} P2 = {f2} P3 = {f1,f3} P3 = {f1,f3}

S1 S2

S4

S1 S2

S4 S1 S2

S3 S5

S1 S2

S3 S5

… …

Pro

ducts

P

roducts

Test

Models

Test

Models

… …

Generate test cases for every product individually is inefficient!

Generate test cases for every product individually is inefficient!

Test

Suites

Test

Suites

S1

S3

S2 S1

S3

S2

[Cichos et al. 2011]

Page 4: Model-based Test Case Generation for (Dynamic) Software … · 2014-05-08 · Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of

ES – Real-Time Systems Lab 4 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines

Testing Software Product Lines

P1 = {f1} P1 = {f1} P2 = {f2} P2 = {f2} P3 = {f1,f3} P3 = {f1,f3}

S1 S2

S4

S1 S2

S4 S1 S2

S3 S5

S1 S2

S3 S5

… …

Pro

ducts

P

roducts

Test

Models

Test

Models

… …

150%

Test

Model

150%

Test

Model

S1 S2

S3 S5

S4

f1

f2

f1 & f3

S1 S2

S3 S5

S4

f1 f1

f2 f2

f1 & f3 f1 & f3

Test

Suites

Test

Suites

S1

S3

S2 S1

S3

S2

[Cichos et al. 2011]

Page 5: Model-based Test Case Generation for (Dynamic) Software … · 2014-05-08 · Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of

ES – Real-Time Systems Lab 5 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines

Testing Software Product Lines

P1 = {f1} P1 = {f1} P2 = {f2} P2 = {f2} P3 = {f1,f3} P3 = {f1,f3}

S1 S2

S4

S1 S2

S4 S1 S2

S3 S5

S1 S2

S3 S5

… …

Pro

ducts

P

roducts

Test

Models

Test

Models

… …

150%

Test

Model

150%

Test

Model

S1 S2

S3 S5

S4

f1

f2

f1 & f3

S1 S2

S3 S5

S4

f1 f1

f2 f2

f1 & f3 f1 & f3

Test

Suites

Test

Suites

S1

S3

S2 S1

S3

S2

[Cichos et al. 2011]

ISO 9001, ISO 26262, ISO 13485 require

certain model coverage for test case generation

Page 6: Model-based Test Case Generation for (Dynamic) Software … · 2014-05-08 · Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of

ES – Real-Time Systems Lab 6 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines

SPL Test Suite Generation

Feature Model Feature Model 150% Test Model

SPL Test Suite (State Coverage)

Config. / TC S1-S2-S3 S1-S2-S4 S1-S2-S3-S5

(r, f1) X

(r, f2) X

(r, f1, f3) X X

S1 S2

S3 S5

S4

f1 f1

f2 f2

f1 & f3 f1 & f3 exclude

f3 f3

r r

f1 f1 f2 f2

[Cichos et al. 2011], [Cichos et al. 2012], [Lochau et al. 2014]

Page 7: Model-based Test Case Generation for (Dynamic) Software … · 2014-05-08 · Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of

ES – Real-Time Systems Lab 7 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines

Using Symbolic Model Checking

for Test Case Generation

S1 S2

S3

S5

S4

S6

f1 f1

f2 f2

Generate C-Code from Model and enrich it with more C-Code

1. S1: int x = y + 10;

2. … // more code

3. S2: if (y < 15) {

4. if (f1)

5. S3: doSth1();

6. else if (f2)

7. S4: doSth2();

8. else goto S6;

9. S5: doSth3();

10. }

11. S6: finish();

Page 8: Model-based Test Case Generation for (Dynamic) Software … · 2014-05-08 · Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of

ES – Real-Time Systems Lab 8 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines

Using Symbolic Model Checking

for Test Case Generation

S1 S2

S3

S5

S4

S6

f1 f1

f2 f2

f1 f2

root

Model-Checker CPAchecker Model-Checker CPAchecker

Varia

bility

Impl.

Achieve state coverage

for test model

1. S1: int x = y + 10;

2. … // more code

3. S2: if (y < 15) {

4. if (f1)

5. S3: doSth1();

6. else if (f2)

7. S4: doSth2();

8. else goto S6;

9. S5: doSth3();

10. }

11. S6: finish();

Page 9: Model-based Test Case Generation for (Dynamic) Software … · 2014-05-08 · Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of

ES – Real-Time Systems Lab 9 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines

Test Case Derivation from Counterexamples

1. S1: int x = y + 10;

2. … // more code

3. S2: if (y < 15) {

4. if (f1)

5. S3: doSth1();

6. else if (f2)

7. S4: doSth2();

8. else goto S6;

9. S5: doSth3();

10. }

11. S6: finish();

1 1

2

3 3

11 11

4

5 5 6

7 7

9 9

(r,f1) (r,f2)

S1 TC1 TC1

S2 TC1 TC1

S3 TC3

S4 TC2

S5 TC3 TC2

S6 TC1 TC1

Test Cases

TC1 y=5, x=15 ⊥

TC2 y=11, x=1 ¬f1⋀f2

TC3 y=11, x=1 f1∧ ¬f2

Assignment

Assumption

Path Condition

Path Condition

Path Condition

[Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013]

Page 10: Model-based Test Case Generation for (Dynamic) Software … · 2014-05-08 · Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of

ES – Real-Time Systems Lab 10 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines

Test Case Derivation from Counterexamples

1. S1: int x = y + 10;

2. … // more code

3. S2: if (y < 15) {

4. if (f1)

5. S3: doSth1();

6. else if (f2)

7. S4: doSth2();

8. else goto S6;

9. S5: doSth3();

10. }

11. S6: finish();

1 1

2

3 3

11 11

4

5 5 6

7 7

9 9

𝑎𝑠𝑠𝑖𝑔𝑛(𝑥 = 𝑦 + 10)

𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 ≥ 15) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 < 15)

𝑎𝑠𝑠𝑢𝑚𝑒(¬𝑓1) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑓1)

𝑎𝑠𝑠𝑢𝑚𝑒(𝑓2)

(r,f1) (r,f2)

S1 TC1 TC1

S2 TC1 TC1

S3 TC3

S4 TC2

S5 TC3 TC2

S6 TC1 TC1

Test Cases

TC1 y=5, x=15 ⊥

TC2 y=11, x=1 ¬f1⋀f2

TC3 y=11, x=1 f1∧ ¬f2 [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013]

Page 11: Model-based Test Case Generation for (Dynamic) Software … · 2014-05-08 · Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of

ES – Real-Time Systems Lab 11 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines

Test Case Derivation from Counterexamples

1. S1: int x = y + 10;

2. … // more code

3. S2: if (y < 15) {

4. if (f1)

5. S3: doSth1();

6. else if (f2)

7. S4: doSth2();

8. else goto S6;

9. S5: doSth3();

10. }

11. S6: finish();

1 1

2

3 3

11 11

4

5 5 6

7 7

9 9

𝑎𝑠𝑠𝑖𝑔𝑛(𝑥 = 𝑦 + 10)

𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)

𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 ≥ 15) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 < 15)

𝑥 = 𝑦 + 10 ∧ 𝑥 ≥ 15 ∧ ((𝑓1 ∧ ...)

𝑎𝑠𝑠𝑢𝑚𝑒(¬𝑓1) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑓1)

𝑎𝑠𝑠𝑢𝑚𝑒(𝑓2)

(r,f1) (r,f2)

S1 TC1 TC1

S2 TC1 TC1

S3 TC3

S4 TC2

S5 TC3 TC2

S6 TC1 TC1

SMT Solver: y = 5 x = 15 f1 = f2 = ⊥

Test Cases

TC1 y=5, x=15 ⊥

TC2 y=11, x=1 ¬f1⋀f2

TC3 y=11, x=1 f1∧ ¬f2 [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013]

𝑡𝑟𝑢𝑒 ∧ ((𝑓1 ∧ ¬𝑓2) ∨ (¬𝑓1 ∧ 𝑓2))

𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)

Page 12: Model-based Test Case Generation for (Dynamic) Software … · 2014-05-08 · Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of

ES – Real-Time Systems Lab 12 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines

Test Case Derivation from Counterexamples

1. S1: int x = y + 10;

2. … // more code

3. S2: if (y < 15) {

4. if (f1)

5. S3: doSth1();

6. else if (f2)

7. S4: doSth2();

8. else goto S6;

9. S5: doSth3();

10. }

11. S6: finish();

1 1

2

3 3

11 11

4

5 5 6

7 7

9 9

𝑎𝑠𝑠𝑖𝑔𝑛(𝑥 = 𝑦 + 10)

𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)

𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 ≥ 15) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 < 15)

𝑥 = 𝑦 + 10 ∧ 𝑥 ≥ 15 ∧ ((𝑓1 ∧ ...)

𝑎𝑠𝑠𝑢𝑚𝑒(¬𝑓1) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑓1)

𝑎𝑠𝑠𝑢𝑚𝑒(𝑓2)

(r,f1) (r,f2)

S1 TC1 TC1

S2 TC1 TC1

S3 TC3

S4 TC2

S5 TC3 TC2

S6 TC1 TC1

SMT Solver: y = 5 x = 15 f1 = f2 = ⊥

Test Cases

TC1 y=5, x=15 ⊥

TC2 y=11, x=1 ¬f1⋀f2

TC3 y=11, x=1 f1∧ ¬f2 [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013]

𝑡𝑟𝑢𝑒 ∧ ((𝑓1 ∧ ¬𝑓2) ∨ (¬𝑓1 ∧ 𝑓2))

𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)

Page 13: Model-based Test Case Generation for (Dynamic) Software … · 2014-05-08 · Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of

ES – Real-Time Systems Lab 13 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines

Test Case Derivation from Counterexamples

1. S1: int x = y + 10;

2. … // more code

3. S2: if (y < 15) {

4. if (f1)

5. S3: doSth1();

6. else if (f2)

7. S4: doSth2();

8. else goto S6;

9. S5: doSth3();

10. }

11. S6: finish();

1 1

2

3 3

11 11

4

5 5 6

7 7

9 9

𝑎𝑠𝑠𝑖𝑔𝑛(𝑥 = 𝑦 + 10)

𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)

𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 ≥ 15) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 < 15)

𝑥 = 𝑦 + 10 ∧ 𝑥 ≥ 15 ∧ ((𝑓1 ∧ ...)

(𝑥 = 𝑦 + 10) ∧ (𝑥 < 15) ∧ ((𝑓1 ∧ ...)

𝑎𝑠𝑠𝑢𝑚𝑒(¬𝑓1) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑓1)

𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2)

𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2)

𝑎𝑠𝑠𝑢𝑚𝑒(𝑓2)

𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2) (r,f1) (r,f2)

S1 TC1 TC1

S2 TC1 TC1

S3 TC3

S4 TC2

S5 TC3 TC2

S6 TC1 TC1

Test Cases

TC1 y=5, x=15 ⊥

TC2 y=11, x=1 ¬f1⋀f2

TC3 y=11, x=1 f1∧ ¬f2 [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013]

𝑡𝑟𝑢𝑒 ∧ ((𝑓1 ∧ ¬𝑓2) ∨ (¬𝑓1 ∧ 𝑓2))

𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)

Page 14: Model-based Test Case Generation for (Dynamic) Software … · 2014-05-08 · Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of

ES – Real-Time Systems Lab 14 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines

Test Case Derivation from Counterexamples

1. S1: int x = y + 10;

2. … // more code

3. S2: if (y < 15) {

4. if (f1)

5. S3: doSth1();

6. else if (f2)

7. S4: doSth2();

8. else goto S6;

9. S5: doSth3();

10. }

11. S6: finish();

1 1

2

3 3

11 11

4

5 5 6

7 7

9 9

𝑎𝑠𝑠𝑖𝑔𝑛(𝑥 = 𝑦 + 10)

𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)

𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 ≥ 15) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 < 15)

𝑥 = 𝑦 + 10 ∧ 𝑥 ≥ 15 ∧ ((𝑓1 ∧ ...)

(𝑥 = 𝑦 + 10) ∧ (𝑥 < 15) ∧ ((𝑓1 ∧ ...)

𝑎𝑠𝑠𝑢𝑚𝑒(¬𝑓1) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑓1)

𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2)

𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2)

𝑎𝑠𝑠𝑢𝑚𝑒(𝑓2)

𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2) (r,f1) (r,f2)

S1 TC1 TC1

S2 TC1 TC1

S3 TC3

S4 TC2

S5 TC3 TC2

S6 TC1 TC1

Test Cases

TC1 y=5, x=15 ⊥

TC2 y=11, x=1 ¬f1⋀f2

TC3 y=11, x=1 f1∧ ¬f2

SMT Solver: y = 1 x = 11 f1 = false f2 = true

[Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013]

𝑡𝑟𝑢𝑒 ∧ ((𝑓1 ∧ ¬𝑓2) ∨ (¬𝑓1 ∧ 𝑓2))

𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)

Page 15: Model-based Test Case Generation for (Dynamic) Software … · 2014-05-08 · Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of

ES – Real-Time Systems Lab 15 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines

Test Case Derivation from Counterexamples

1. S1: int x = y + 10;

2. … // more code

3. S2: if (y < 15) {

4. if (f1)

5. S3: doSth1();

6. else if (f2)

7. S4: doSth2();

8. else goto S6;

9. S5: doSth3();

10. }

11. S6: finish();

1 1

2

3 3

11 11

4

5 5 6

7 7

9 9

𝑎𝑠𝑠𝑖𝑔𝑛(𝑥 = 𝑦 + 10)

𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)

𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 ≥ 15) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 < 15)

𝑥 = 𝑦 + 10 ∧ 𝑥 ≥ 15 ∧ ((𝑓1 ∧ ...)

(𝑥 = 𝑦 + 10) ∧ (𝑥 < 15) ∧ ((𝑓1 ∧ ...)

𝑎𝑠𝑠𝑢𝑚𝑒(¬𝑓1) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑓1)

𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (𝑓1 ∧ ¬𝑓2)

𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2)

𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2)

𝑎𝑠𝑠𝑢𝑚𝑒(𝑓2)

𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2) (r,f1) (r,f2)

S1 TC1 TC1

S2 TC1 TC1

S3 TC3

S4 TC2

S5 TC3 TC2

S6 TC1 TC1

Test Cases

TC1 y=5, x=15 ⊥

TC2 y=11, x=1 ¬f1⋀f2

TC3 y=11, x=1 f1∧ ¬f2 [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013]

𝑡𝑟𝑢𝑒 ∧ ((𝑓1 ∧ ¬𝑓2) ∨ (¬𝑓1 ∧ 𝑓2))

𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)

Page 16: Model-based Test Case Generation for (Dynamic) Software … · 2014-05-08 · Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of

ES – Real-Time Systems Lab 16 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines

Test Case Derivation from Counterexamples

1. S1: int x = y + 10;

2. … // more code

3. S2: if (y < 15) {

4. if (f1)

5. S3: doSth1();

6. else if (f2)

7. S4: doSth2();

8. else goto S6;

9. S5: doSth3();

10. }

11. S6: finish();

1 1

2

3 3

11 11

4

5 5 6

7 7

9 9

𝑎𝑠𝑠𝑖𝑔𝑛(𝑥 = 𝑦 + 10)

𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)

𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 ≥ 15) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 < 15)

𝑥 = 𝑦 + 10 ∧ 𝑥 ≥ 15 ∧ ((𝑓1 ∧ ...)

(𝑥 = 𝑦 + 10) ∧ (𝑥 < 15) ∧ ((𝑓1 ∧ ...)

𝑎𝑠𝑠𝑢𝑚𝑒(¬𝑓1) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑓1)

𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (𝑓1 ∧ ¬𝑓2)

𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2)

𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2)

𝑎𝑠𝑠𝑢𝑚𝑒(𝑓2)

𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2) (r,f1) (r,f2)

S1 TC1 TC1

S2 TC1 TC1

S3 TC3

S4 TC2

S5 TC3 TC2

S6 TC1 TC1

Test Cases

TC1 y=5, x=15 ⊥

TC2 y=11, x=1 ¬f1⋀f2

TC3 y=11, x=1 f1∧ ¬f2

SMT Solver: y = 1 x = 11 f1 = true f2 = false

[Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013]

𝑡𝑟𝑢𝑒 ∧ ((𝑓1 ∧ ¬𝑓2) ∨ (¬𝑓1 ∧ 𝑓2))

𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)

Page 17: Model-based Test Case Generation for (Dynamic) Software … · 2014-05-08 · Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of

ES – Real-Time Systems Lab 17 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines

Test Case Derivation from Counterexamples

1. S1: int x = y + 10;

2. … // more code

3. S2: if (y < 15) {

4. if (f1)

5. S3: doSth1();

6. else if (f2)

7. S4: doSth2();

8. else goto S6;

9. S5: doSth3();

10. }

11. S6: finish();

1 1

2

3 3

11 11

4

5 5 6

7 7

9 9

𝑎𝑠𝑠𝑖𝑔𝑛(𝑥 = 𝑦 + 10)

𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)

𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 ≥ 15) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 < 15)

𝑥 = 𝑦 + 10 ∧ 𝑥 ≥ 15 ∧ ((𝑓1 ∧ ...)

(𝑥 = 𝑦 + 10) ∧ (𝑥 < 15) ∧ ((𝑓1 ∧ ...)

𝑎𝑠𝑠𝑢𝑚𝑒(¬𝑓1) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑓1)

𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (𝑓1 ∧ ¬𝑓2)

𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2)

𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2)

𝑎𝑠𝑠𝑢𝑚𝑒(𝑓2)

𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2) (r,f1) (r,f2)

S1 TC1 TC1

S2 TC1 TC1

S3 TC3

S4 TC2

S5 TC3 TC2

S6 TC1 TC1

Test Cases

TC1 y=5, x=15 ⊥

TC2 y=11, x=1 ¬f1⋀f2

TC3 y=11, x=1 f1∧ ¬f2

S5 may also be covered in

the variant (r,f1)

[Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013]

A BDD analysis checks if a location may be reached by a path with another

feature configuration.

A BDD analysis checks if a location may be reached by a path with another

feature configuration.

𝑡𝑟𝑢𝑒 ∧ ((𝑓1 ∧ ¬𝑓2) ∨ (¬𝑓1 ∧ 𝑓2))

𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)

Page 18: Model-based Test Case Generation for (Dynamic) Software … · 2014-05-08 · Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of

ES – Real-Time Systems Lab 18 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines

𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ ((¬𝑓1 ∧ 𝑓2) ∨ (𝑓1 ∧ ¬𝑓2))

Test Case Derivation from Counterexamples

1. S1: int x = y + 10;

2. … // more code

3. S2: if (y < 15) {

4. if (f1)

5. S3: doSth1();

6. else if (f2)

7. S4: doSth2();

8. else goto S6;

9. S5: doSth3();

10. }

11. S6: finish();

1 1

2

3 3

11 11

4

5 5 6

7 7

9 9

𝑎𝑠𝑠𝑖𝑔𝑛(𝑥 = 𝑦 + 10)

𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)

𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 ≥ 15) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 < 15)

𝑥 = 𝑦 + 10 ∧ 𝑥 ≥ 15 ∧ ((𝑓1 ∧ ...)

(𝑥 = 𝑦 + 10) ∧ (𝑥 < 15) ∧ ((𝑓1 ∧ ...)

𝑎𝑠𝑠𝑢𝑚𝑒(¬𝑓1) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑓1)

𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (𝑓1 ∧ ¬𝑓2)

𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2)

𝑎𝑠𝑠𝑢𝑚𝑒(𝑓2)

𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2) (r,f1) (r,f2)

S1 TC1 TC1

S2 TC1 TC1

S3 TC3

S4 TC2

S5 TC3 TC2

S6 TC1 TC1

Test Cases

TC1 y=5, x=15 ⊥

TC2 y=11, x=1 ¬f1⋀f2

TC3 y=11, x=1 f1∧ ¬f2 [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013]

𝑡𝑟𝑢𝑒 ∧ ((𝑓1 ∧ ¬𝑓2) ∨ (¬𝑓1 ∧ 𝑓2))

𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)

Page 19: Model-based Test Case Generation for (Dynamic) Software … · 2014-05-08 · Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of

ES – Real-Time Systems Lab 19 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines

𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ ((¬𝑓1 ∧ 𝑓2) ∨ (𝑓1 ∧ ¬𝑓2))

Test Case Derivation from Counterexamples

1. S1: int x = y + 10;

2. … // more code

3. S2: if (y < 15) {

4. if (f1)

5. S3: doSth1();

6. else if (f2)

7. S4: doSth2();

8. else goto S6;

9. S5: doSth3();

10. }

11. S6: finish();

1 1

2

3 3

11 11

4

5 5 6

7 7

9 9

𝑎𝑠𝑠𝑖𝑔𝑛(𝑥 = 𝑦 + 10)

𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)

𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 ≥ 15) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 < 15)

𝑥 = 𝑦 + 10 ∧ 𝑥 ≥ 15 ∧ ((𝑓1 ∧ ...)

(𝑥 = 𝑦 + 10) ∧ (𝑥 < 15) ∧ ((𝑓1 ∧ ...)

𝑎𝑠𝑠𝑢𝑚𝑒(¬𝑓1) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑓1)

𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (𝑓1 ∧ ¬𝑓2)

𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2)

𝑎𝑠𝑠𝑢𝑚𝑒(𝑓2)

𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2) (r,f1) (r,f2)

S1 TC1 TC1

S2 TC1 TC1

S3 TC3

S4 TC2

S5 TC3 TC2

S6 TC1 TC1

Test Cases

TC1 y=5, x=15 ⊥

TC2 y=11, x=1 ¬f1⋀f2

TC3 y=11, x=1 f1∧ ¬f2 [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013]

𝑡𝑟𝑢𝑒 ∧ ((𝑓1 ∧ ¬𝑓2) ∨ (¬𝑓1 ∧ 𝑓2))

𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)

Page 20: Model-based Test Case Generation for (Dynamic) Software … · 2014-05-08 · Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of

ES – Real-Time Systems Lab 20 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines

Testing Dynamic Software Product Lines

- Testing SPLs: test that every derivable variant is correctly implemented

- Testing DSPLs: test that all possible reconfigurations are correct

S1

S2

S3

1. S1: int x = y +

10;

2. … // more code

3. S2: if (y < 15) {

4. if (f1)

5. S3:

doSth1();

6. else if (f2)

7. S4:

doSth2();

8. else goto S6;

9. S5: doSth3();

10. }

11. S6: finish();

Environment Environment Program Program

Non-deterministic behavior C1

C3

C2

Reconfiguration

Non-deterministic influence on

feature variables

Non-deterministic influence on

program variables

Page 21: Model-based Test Case Generation for (Dynamic) Software … · 2014-05-08 · Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of

ES – Real-Time Systems Lab 21 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines

Testing Dynamic Software Product Lines

- Testing SPLs: test that every derivable variant is correctly implemented

- Testing DSPLs: test that all possible reconfigurations are correct

- CPAchecker offers techniques to handle non-deterministic behavior

S1

S2

S3

1. S1: int x = y +

10;

2. … // more code

3. S2: if (y < 15) {

4. if (f1)

5. S3:

doSth1();

6. else if (f2)

7. S4:

doSth2();

8. else goto S6;

9. S5: doSth3();

10. }

11. S6: finish();

Environment Environment Program Program

Non-deterministic behavior C1

C3

C2

Reconfiguration

Page 22: Model-based Test Case Generation for (Dynamic) Software … · 2014-05-08 · Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of

ES – Real-Time Systems Lab 22 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines

Testing Dynamic Software Product Lines

- Testing SPLs: test that every derivable variant is correctly implemented

- Testing DSPLs: test that all possible reconfigurations are correct

- CPAchecker offers techniques to handle non-deterministic behavior

S1

S2

S3

1. S1: int x = y +

10;

2. … // more code

3. S2: if (y < 15) {

4. if (f1)

5. S3:

doSth1();

6. else if (f2)

7. S4:

doSth2();

8. else goto S6;

9. S5: doSth3();

10. }

11. S6: finish();

Environment Environment Program Program

Non-deterministic behavior C1

C3

C2

Reconfiguration

1. How to test reconfigurations efficiently? 1. How to test reconfigurations efficiently?

2. What is a suitable coverage criteria? 2. What is a suitable coverage criteria?

3. How to ensure liveness of the reconfiguration life cycle? 3. How to ensure liveness of the reconfiguration life cycle?

Page 23: Model-based Test Case Generation for (Dynamic) Software … · 2014-05-08 · Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of

ES – Real-Time Systems Lab 23 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines

List of Literature

[Cichos et al. 2011] H. Cichos, S. Oster, M. Lochau, A. Schürr : Model-Based Coverage-

Driven Test Suite Generation for Software Product Lines. MoDELS 2011:

425-439

[Cichos et al. 2012] H. Cichos, M. Lochau, S. Oster, A. Schürr : Reduktion von Testsuiten für

Software-Produktlinien. Software Engineering 2012: 143-154

[Lochau et al. 2014] M. Lochau, J. Bürdek, S. Lity, M. Hagner, C. Legat, U. Goltz, A. Schürr :

Applying Model-based Software Product Line Testing Approaches to the

Automation Engineering Domain. AT Journal 2014 (submitted)

[Holzer et al. 2011] A. Holzer, M. Tautschnig, C. Schallhart, H. Veith : An Introduction to Test

Specification in FQL. In S. Barner, I. Harris, D. Kroening, and O. Raz,

editors, Hardware and Software: Verification and Testing, volume 6504 of

Lecture Notes in Computer Science, pages 9–22. Springer Berlin

Heidelberg, 2011.

[Beyer et al. 2011] D. Beyer and M. E. Keremoglu : CPAchecker: A Tool for Configurable

Software Verification. In Proc. CAV, pages 184-190, Springer, 2011.

[Holzmann 1997] G. J. Holzmann : The Model Checker Spin, IEEE Trans. on Software

Engineering, Vol. 23, No. 5, May 1997, pp. 279-295.

Page 24: Model-based Test Case Generation for (Dynamic) Software … · 2014-05-08 · Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of

ES – Real-Time Systems Lab 24 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines

List of Literature (2/2)

[Beyer et al. 2004] D. Beyer, A. J. Chlipala, T. A. Henzinger, R. Jhala, and R. Majumdar.

Generating Test from Counterexamples. In Proc. of the 26th ICSE, pages

326–335, 2004.

[Rhein et al. 2011] A. von Rhein, S. Apel, F. Raimondi : Introducing Binary Decision

Diagrams in the Explicit-State Verification of Java Code. Java Pathfinder

Workshop (co-located with ASE'11), 2011.

[Apel et al. 2013] S. Apel, A. von Rhein, P. Wendler, A. Größlinger, D. Beyer : Strategies

for Product-Line Verification: Case Studies and Experiments. In

Proceedings of the IEEE/ACM International Conference on Software

Engineering (ICSE), pages 482–491. IEEE Computer Society, 2013.