Upload
others
View
6
Download
0
Embed Size (px)
Citation preview
© author(s) of these slides 2008 including research results of the research network ES and TU Darmstadt otherwise as specified at the respective slide
08.05.2014
Johannes Bürdek
Tel.+49 6151 16 76089
ES Real-Time Systems Lab
Prof. Dr. rer. nat. Andy Schürr
Dept. of Electrical Engineering and Information Technology
Dept. of Computer Science (adjunct Professor)
www.es.tu-darmstadt.de
Model-based Test Case Generation for
(Dynamic) Software Product Lines
The 6th Meeting on Feature-oriented Software Development 2014, May 04-07 – Schloss Dagstuhl, Germany
DFG Priority Programme 1593 Design for Future – Managed Software Evolution
ES – Real-Time Systems Lab 2 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines
Testing Software Product Lines
P1 = {f1} P1 = {f1} P2 = {f2} P2 = {f2} P3 = {f1,f3} P3 = {f1,f3}
S1 S2
S4
S1 S2
S4 S1 S2
S3 S5
S1 S2
S3 S5
… …
Pro
ducts
P
roducts
Test
Models
Test
Models
… …
Test
Suites
Test
Suites
S1
S3
S2 S1
S3
S2
[Cichos et al. 2011]
ES – Real-Time Systems Lab 3 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines
Testing Software Product Lines
P1 = {f1} P1 = {f1} P2 = {f2} P2 = {f2} P3 = {f1,f3} P3 = {f1,f3}
S1 S2
S4
S1 S2
S4 S1 S2
S3 S5
S1 S2
S3 S5
… …
Pro
ducts
P
roducts
Test
Models
Test
Models
… …
Generate test cases for every product individually is inefficient!
Generate test cases for every product individually is inefficient!
Test
Suites
Test
Suites
S1
S3
S2 S1
S3
S2
[Cichos et al. 2011]
ES – Real-Time Systems Lab 4 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines
Testing Software Product Lines
P1 = {f1} P1 = {f1} P2 = {f2} P2 = {f2} P3 = {f1,f3} P3 = {f1,f3}
S1 S2
S4
S1 S2
S4 S1 S2
S3 S5
S1 S2
S3 S5
… …
Pro
ducts
P
roducts
Test
Models
Test
Models
… …
150%
Test
Model
150%
Test
Model
S1 S2
S3 S5
S4
f1
f2
f1 & f3
S1 S2
S3 S5
S4
f1 f1
f2 f2
f1 & f3 f1 & f3
Test
Suites
Test
Suites
S1
S3
S2 S1
S3
S2
[Cichos et al. 2011]
ES – Real-Time Systems Lab 5 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines
Testing Software Product Lines
P1 = {f1} P1 = {f1} P2 = {f2} P2 = {f2} P3 = {f1,f3} P3 = {f1,f3}
S1 S2
S4
S1 S2
S4 S1 S2
S3 S5
S1 S2
S3 S5
… …
Pro
ducts
P
roducts
Test
Models
Test
Models
… …
150%
Test
Model
150%
Test
Model
S1 S2
S3 S5
S4
f1
f2
f1 & f3
S1 S2
S3 S5
S4
f1 f1
f2 f2
f1 & f3 f1 & f3
Test
Suites
Test
Suites
S1
S3
S2 S1
S3
S2
[Cichos et al. 2011]
ISO 9001, ISO 26262, ISO 13485 require
certain model coverage for test case generation
ES – Real-Time Systems Lab 6 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines
SPL Test Suite Generation
Feature Model Feature Model 150% Test Model
SPL Test Suite (State Coverage)
Config. / TC S1-S2-S3 S1-S2-S4 S1-S2-S3-S5
(r, f1) X
(r, f2) X
(r, f1, f3) X X
S1 S2
S3 S5
S4
f1 f1
f2 f2
f1 & f3 f1 & f3 exclude
f3 f3
r r
f1 f1 f2 f2
[Cichos et al. 2011], [Cichos et al. 2012], [Lochau et al. 2014]
ES – Real-Time Systems Lab 7 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines
Using Symbolic Model Checking
for Test Case Generation
S1 S2
S3
S5
S4
S6
f1 f1
f2 f2
Generate C-Code from Model and enrich it with more C-Code
1. S1: int x = y + 10;
2. … // more code
3. S2: if (y < 15) {
4. if (f1)
5. S3: doSth1();
6. else if (f2)
7. S4: doSth2();
8. else goto S6;
9. S5: doSth3();
10. }
11. S6: finish();
ES – Real-Time Systems Lab 8 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines
Using Symbolic Model Checking
for Test Case Generation
S1 S2
S3
S5
S4
S6
f1 f1
f2 f2
f1 f2
root
Model-Checker CPAchecker Model-Checker CPAchecker
Varia
bility
Impl.
Achieve state coverage
for test model
1. S1: int x = y + 10;
2. … // more code
3. S2: if (y < 15) {
4. if (f1)
5. S3: doSth1();
6. else if (f2)
7. S4: doSth2();
8. else goto S6;
9. S5: doSth3();
10. }
11. S6: finish();
ES – Real-Time Systems Lab 9 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines
Test Case Derivation from Counterexamples
1. S1: int x = y + 10;
2. … // more code
3. S2: if (y < 15) {
4. if (f1)
5. S3: doSth1();
6. else if (f2)
7. S4: doSth2();
8. else goto S6;
9. S5: doSth3();
10. }
11. S6: finish();
1 1
2
3 3
11 11
4
5 5 6
7 7
9 9
(r,f1) (r,f2)
S1 TC1 TC1
S2 TC1 TC1
S3 TC3
S4 TC2
S5 TC3 TC2
S6 TC1 TC1
Test Cases
TC1 y=5, x=15 ⊥
TC2 y=11, x=1 ¬f1⋀f2
TC3 y=11, x=1 f1∧ ¬f2
Assignment
Assumption
Path Condition
Path Condition
Path Condition
…
[Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013]
ES – Real-Time Systems Lab 10 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines
Test Case Derivation from Counterexamples
1. S1: int x = y + 10;
2. … // more code
3. S2: if (y < 15) {
4. if (f1)
5. S3: doSth1();
6. else if (f2)
7. S4: doSth2();
8. else goto S6;
9. S5: doSth3();
10. }
11. S6: finish();
1 1
2
3 3
11 11
4
5 5 6
7 7
9 9
𝑎𝑠𝑠𝑖𝑔𝑛(𝑥 = 𝑦 + 10)
𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 ≥ 15) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 < 15)
𝑎𝑠𝑠𝑢𝑚𝑒(¬𝑓1) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑓1)
𝑎𝑠𝑠𝑢𝑚𝑒(𝑓2)
(r,f1) (r,f2)
S1 TC1 TC1
S2 TC1 TC1
S3 TC3
S4 TC2
S5 TC3 TC2
S6 TC1 TC1
Test Cases
TC1 y=5, x=15 ⊥
TC2 y=11, x=1 ¬f1⋀f2
TC3 y=11, x=1 f1∧ ¬f2 [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013]
ES – Real-Time Systems Lab 11 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines
Test Case Derivation from Counterexamples
1. S1: int x = y + 10;
2. … // more code
3. S2: if (y < 15) {
4. if (f1)
5. S3: doSth1();
6. else if (f2)
7. S4: doSth2();
8. else goto S6;
9. S5: doSth3();
10. }
11. S6: finish();
1 1
2
3 3
11 11
4
5 5 6
7 7
9 9
𝑎𝑠𝑠𝑖𝑔𝑛(𝑥 = 𝑦 + 10)
𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)
𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 ≥ 15) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 < 15)
𝑥 = 𝑦 + 10 ∧ 𝑥 ≥ 15 ∧ ((𝑓1 ∧ ...)
𝑎𝑠𝑠𝑢𝑚𝑒(¬𝑓1) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑓1)
𝑎𝑠𝑠𝑢𝑚𝑒(𝑓2)
(r,f1) (r,f2)
S1 TC1 TC1
S2 TC1 TC1
S3 TC3
S4 TC2
S5 TC3 TC2
S6 TC1 TC1
SMT Solver: y = 5 x = 15 f1 = f2 = ⊥
Test Cases
TC1 y=5, x=15 ⊥
TC2 y=11, x=1 ¬f1⋀f2
TC3 y=11, x=1 f1∧ ¬f2 [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013]
𝑡𝑟𝑢𝑒 ∧ ((𝑓1 ∧ ¬𝑓2) ∨ (¬𝑓1 ∧ 𝑓2))
𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)
ES – Real-Time Systems Lab 12 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines
Test Case Derivation from Counterexamples
1. S1: int x = y + 10;
2. … // more code
3. S2: if (y < 15) {
4. if (f1)
5. S3: doSth1();
6. else if (f2)
7. S4: doSth2();
8. else goto S6;
9. S5: doSth3();
10. }
11. S6: finish();
1 1
2
3 3
11 11
4
5 5 6
7 7
9 9
𝑎𝑠𝑠𝑖𝑔𝑛(𝑥 = 𝑦 + 10)
𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)
𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 ≥ 15) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 < 15)
𝑥 = 𝑦 + 10 ∧ 𝑥 ≥ 15 ∧ ((𝑓1 ∧ ...)
𝑎𝑠𝑠𝑢𝑚𝑒(¬𝑓1) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑓1)
𝑎𝑠𝑠𝑢𝑚𝑒(𝑓2)
(r,f1) (r,f2)
S1 TC1 TC1
S2 TC1 TC1
S3 TC3
S4 TC2
S5 TC3 TC2
S6 TC1 TC1
SMT Solver: y = 5 x = 15 f1 = f2 = ⊥
Test Cases
TC1 y=5, x=15 ⊥
TC2 y=11, x=1 ¬f1⋀f2
TC3 y=11, x=1 f1∧ ¬f2 [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013]
𝑡𝑟𝑢𝑒 ∧ ((𝑓1 ∧ ¬𝑓2) ∨ (¬𝑓1 ∧ 𝑓2))
𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)
ES – Real-Time Systems Lab 13 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines
Test Case Derivation from Counterexamples
1. S1: int x = y + 10;
2. … // more code
3. S2: if (y < 15) {
4. if (f1)
5. S3: doSth1();
6. else if (f2)
7. S4: doSth2();
8. else goto S6;
9. S5: doSth3();
10. }
11. S6: finish();
1 1
2
3 3
11 11
4
5 5 6
7 7
9 9
𝑎𝑠𝑠𝑖𝑔𝑛(𝑥 = 𝑦 + 10)
𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)
𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 ≥ 15) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 < 15)
𝑥 = 𝑦 + 10 ∧ 𝑥 ≥ 15 ∧ ((𝑓1 ∧ ...)
(𝑥 = 𝑦 + 10) ∧ (𝑥 < 15) ∧ ((𝑓1 ∧ ...)
𝑎𝑠𝑠𝑢𝑚𝑒(¬𝑓1) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑓1)
𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2)
𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2)
𝑎𝑠𝑠𝑢𝑚𝑒(𝑓2)
𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2) (r,f1) (r,f2)
S1 TC1 TC1
S2 TC1 TC1
S3 TC3
S4 TC2
S5 TC3 TC2
S6 TC1 TC1
Test Cases
TC1 y=5, x=15 ⊥
TC2 y=11, x=1 ¬f1⋀f2
TC3 y=11, x=1 f1∧ ¬f2 [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013]
𝑡𝑟𝑢𝑒 ∧ ((𝑓1 ∧ ¬𝑓2) ∨ (¬𝑓1 ∧ 𝑓2))
𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)
ES – Real-Time Systems Lab 14 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines
Test Case Derivation from Counterexamples
1. S1: int x = y + 10;
2. … // more code
3. S2: if (y < 15) {
4. if (f1)
5. S3: doSth1();
6. else if (f2)
7. S4: doSth2();
8. else goto S6;
9. S5: doSth3();
10. }
11. S6: finish();
1 1
2
3 3
11 11
4
5 5 6
7 7
9 9
𝑎𝑠𝑠𝑖𝑔𝑛(𝑥 = 𝑦 + 10)
𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)
𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 ≥ 15) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 < 15)
𝑥 = 𝑦 + 10 ∧ 𝑥 ≥ 15 ∧ ((𝑓1 ∧ ...)
(𝑥 = 𝑦 + 10) ∧ (𝑥 < 15) ∧ ((𝑓1 ∧ ...)
𝑎𝑠𝑠𝑢𝑚𝑒(¬𝑓1) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑓1)
𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2)
𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2)
𝑎𝑠𝑠𝑢𝑚𝑒(𝑓2)
𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2) (r,f1) (r,f2)
S1 TC1 TC1
S2 TC1 TC1
S3 TC3
S4 TC2
S5 TC3 TC2
S6 TC1 TC1
Test Cases
TC1 y=5, x=15 ⊥
TC2 y=11, x=1 ¬f1⋀f2
TC3 y=11, x=1 f1∧ ¬f2
SMT Solver: y = 1 x = 11 f1 = false f2 = true
[Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013]
𝑡𝑟𝑢𝑒 ∧ ((𝑓1 ∧ ¬𝑓2) ∨ (¬𝑓1 ∧ 𝑓2))
𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)
ES – Real-Time Systems Lab 15 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines
Test Case Derivation from Counterexamples
1. S1: int x = y + 10;
2. … // more code
3. S2: if (y < 15) {
4. if (f1)
5. S3: doSth1();
6. else if (f2)
7. S4: doSth2();
8. else goto S6;
9. S5: doSth3();
10. }
11. S6: finish();
1 1
2
3 3
11 11
4
5 5 6
7 7
9 9
𝑎𝑠𝑠𝑖𝑔𝑛(𝑥 = 𝑦 + 10)
𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)
𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 ≥ 15) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 < 15)
𝑥 = 𝑦 + 10 ∧ 𝑥 ≥ 15 ∧ ((𝑓1 ∧ ...)
(𝑥 = 𝑦 + 10) ∧ (𝑥 < 15) ∧ ((𝑓1 ∧ ...)
𝑎𝑠𝑠𝑢𝑚𝑒(¬𝑓1) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑓1)
𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (𝑓1 ∧ ¬𝑓2)
𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2)
𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2)
𝑎𝑠𝑠𝑢𝑚𝑒(𝑓2)
𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2) (r,f1) (r,f2)
S1 TC1 TC1
S2 TC1 TC1
S3 TC3
S4 TC2
S5 TC3 TC2
S6 TC1 TC1
Test Cases
TC1 y=5, x=15 ⊥
TC2 y=11, x=1 ¬f1⋀f2
TC3 y=11, x=1 f1∧ ¬f2 [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013]
𝑡𝑟𝑢𝑒 ∧ ((𝑓1 ∧ ¬𝑓2) ∨ (¬𝑓1 ∧ 𝑓2))
𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)
ES – Real-Time Systems Lab 16 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines
Test Case Derivation from Counterexamples
1. S1: int x = y + 10;
2. … // more code
3. S2: if (y < 15) {
4. if (f1)
5. S3: doSth1();
6. else if (f2)
7. S4: doSth2();
8. else goto S6;
9. S5: doSth3();
10. }
11. S6: finish();
1 1
2
3 3
11 11
4
5 5 6
7 7
9 9
𝑎𝑠𝑠𝑖𝑔𝑛(𝑥 = 𝑦 + 10)
𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)
𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 ≥ 15) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 < 15)
𝑥 = 𝑦 + 10 ∧ 𝑥 ≥ 15 ∧ ((𝑓1 ∧ ...)
(𝑥 = 𝑦 + 10) ∧ (𝑥 < 15) ∧ ((𝑓1 ∧ ...)
𝑎𝑠𝑠𝑢𝑚𝑒(¬𝑓1) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑓1)
𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (𝑓1 ∧ ¬𝑓2)
𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2)
𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2)
𝑎𝑠𝑠𝑢𝑚𝑒(𝑓2)
𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2) (r,f1) (r,f2)
S1 TC1 TC1
S2 TC1 TC1
S3 TC3
S4 TC2
S5 TC3 TC2
S6 TC1 TC1
Test Cases
TC1 y=5, x=15 ⊥
TC2 y=11, x=1 ¬f1⋀f2
TC3 y=11, x=1 f1∧ ¬f2
SMT Solver: y = 1 x = 11 f1 = true f2 = false
[Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013]
𝑡𝑟𝑢𝑒 ∧ ((𝑓1 ∧ ¬𝑓2) ∨ (¬𝑓1 ∧ 𝑓2))
𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)
ES – Real-Time Systems Lab 17 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines
Test Case Derivation from Counterexamples
1. S1: int x = y + 10;
2. … // more code
3. S2: if (y < 15) {
4. if (f1)
5. S3: doSth1();
6. else if (f2)
7. S4: doSth2();
8. else goto S6;
9. S5: doSth3();
10. }
11. S6: finish();
1 1
2
3 3
11 11
4
5 5 6
7 7
9 9
𝑎𝑠𝑠𝑖𝑔𝑛(𝑥 = 𝑦 + 10)
𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)
𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 ≥ 15) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 < 15)
𝑥 = 𝑦 + 10 ∧ 𝑥 ≥ 15 ∧ ((𝑓1 ∧ ...)
(𝑥 = 𝑦 + 10) ∧ (𝑥 < 15) ∧ ((𝑓1 ∧ ...)
𝑎𝑠𝑠𝑢𝑚𝑒(¬𝑓1) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑓1)
𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (𝑓1 ∧ ¬𝑓2)
𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2)
𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2)
𝑎𝑠𝑠𝑢𝑚𝑒(𝑓2)
𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2) (r,f1) (r,f2)
S1 TC1 TC1
S2 TC1 TC1
S3 TC3
S4 TC2
S5 TC3 TC2
S6 TC1 TC1
Test Cases
TC1 y=5, x=15 ⊥
TC2 y=11, x=1 ¬f1⋀f2
TC3 y=11, x=1 f1∧ ¬f2
S5 may also be covered in
the variant (r,f1)
[Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013]
A BDD analysis checks if a location may be reached by a path with another
feature configuration.
A BDD analysis checks if a location may be reached by a path with another
feature configuration.
𝑡𝑟𝑢𝑒 ∧ ((𝑓1 ∧ ¬𝑓2) ∨ (¬𝑓1 ∧ 𝑓2))
𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)
ES – Real-Time Systems Lab 18 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines
𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ ((¬𝑓1 ∧ 𝑓2) ∨ (𝑓1 ∧ ¬𝑓2))
Test Case Derivation from Counterexamples
1. S1: int x = y + 10;
2. … // more code
3. S2: if (y < 15) {
4. if (f1)
5. S3: doSth1();
6. else if (f2)
7. S4: doSth2();
8. else goto S6;
9. S5: doSth3();
10. }
11. S6: finish();
1 1
2
3 3
11 11
4
5 5 6
7 7
9 9
𝑎𝑠𝑠𝑖𝑔𝑛(𝑥 = 𝑦 + 10)
𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)
𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 ≥ 15) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 < 15)
𝑥 = 𝑦 + 10 ∧ 𝑥 ≥ 15 ∧ ((𝑓1 ∧ ...)
(𝑥 = 𝑦 + 10) ∧ (𝑥 < 15) ∧ ((𝑓1 ∧ ...)
𝑎𝑠𝑠𝑢𝑚𝑒(¬𝑓1) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑓1)
𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (𝑓1 ∧ ¬𝑓2)
𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2)
𝑎𝑠𝑠𝑢𝑚𝑒(𝑓2)
𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2) (r,f1) (r,f2)
S1 TC1 TC1
S2 TC1 TC1
S3 TC3
S4 TC2
S5 TC3 TC2
S6 TC1 TC1
Test Cases
TC1 y=5, x=15 ⊥
TC2 y=11, x=1 ¬f1⋀f2
TC3 y=11, x=1 f1∧ ¬f2 [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013]
𝑡𝑟𝑢𝑒 ∧ ((𝑓1 ∧ ¬𝑓2) ∨ (¬𝑓1 ∧ 𝑓2))
𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)
ES – Real-Time Systems Lab 19 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines
𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ ((¬𝑓1 ∧ 𝑓2) ∨ (𝑓1 ∧ ¬𝑓2))
Test Case Derivation from Counterexamples
1. S1: int x = y + 10;
2. … // more code
3. S2: if (y < 15) {
4. if (f1)
5. S3: doSth1();
6. else if (f2)
7. S4: doSth2();
8. else goto S6;
9. S5: doSth3();
10. }
11. S6: finish();
1 1
2
3 3
11 11
4
5 5 6
7 7
9 9
𝑎𝑠𝑠𝑖𝑔𝑛(𝑥 = 𝑦 + 10)
𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)
𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 ≥ 15) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑥 < 15)
𝑥 = 𝑦 + 10 ∧ 𝑥 ≥ 15 ∧ ((𝑓1 ∧ ...)
(𝑥 = 𝑦 + 10) ∧ (𝑥 < 15) ∧ ((𝑓1 ∧ ...)
𝑎𝑠𝑠𝑢𝑚𝑒(¬𝑓1) 𝑎𝑠𝑠𝑢𝑚𝑒(𝑓1)
𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (𝑓1 ∧ ¬𝑓2)
𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2)
𝑎𝑠𝑠𝑢𝑚𝑒(𝑓2)
𝑥 = 𝑦 + 10 ∧ 𝑥 < 15 ∧ (¬𝑓1 ∧ 𝑓2) (r,f1) (r,f2)
S1 TC1 TC1
S2 TC1 TC1
S3 TC3
S4 TC2
S5 TC3 TC2
S6 TC1 TC1
Test Cases
TC1 y=5, x=15 ⊥
TC2 y=11, x=1 ¬f1⋀f2
TC3 y=11, x=1 f1∧ ¬f2 [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013]
𝑡𝑟𝑢𝑒 ∧ ((𝑓1 ∧ ¬𝑓2) ∨ (¬𝑓1 ∧ 𝑓2))
𝑥 = 𝑦 + 10 ∧ ((𝑓1 ∧ ...)
⊥
ES – Real-Time Systems Lab 20 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines
Testing Dynamic Software Product Lines
- Testing SPLs: test that every derivable variant is correctly implemented
- Testing DSPLs: test that all possible reconfigurations are correct
S1
S2
S3
1. S1: int x = y +
10;
2. … // more code
3. S2: if (y < 15) {
4. if (f1)
5. S3:
doSth1();
6. else if (f2)
7. S4:
doSth2();
8. else goto S6;
9. S5: doSth3();
10. }
11. S6: finish();
Environment Environment Program Program
Non-deterministic behavior C1
C3
C2
Reconfiguration
Non-deterministic influence on
feature variables
Non-deterministic influence on
program variables
ES – Real-Time Systems Lab 21 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines
Testing Dynamic Software Product Lines
- Testing SPLs: test that every derivable variant is correctly implemented
- Testing DSPLs: test that all possible reconfigurations are correct
- CPAchecker offers techniques to handle non-deterministic behavior
S1
S2
S3
1. S1: int x = y +
10;
2. … // more code
3. S2: if (y < 15) {
4. if (f1)
5. S3:
doSth1();
6. else if (f2)
7. S4:
doSth2();
8. else goto S6;
9. S5: doSth3();
10. }
11. S6: finish();
Environment Environment Program Program
Non-deterministic behavior C1
C3
C2
Reconfiguration
ES – Real-Time Systems Lab 22 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines
Testing Dynamic Software Product Lines
- Testing SPLs: test that every derivable variant is correctly implemented
- Testing DSPLs: test that all possible reconfigurations are correct
- CPAchecker offers techniques to handle non-deterministic behavior
S1
S2
S3
1. S1: int x = y +
10;
2. … // more code
3. S2: if (y < 15) {
4. if (f1)
5. S3:
doSth1();
6. else if (f2)
7. S4:
doSth2();
8. else goto S6;
9. S5: doSth3();
10. }
11. S6: finish();
Environment Environment Program Program
Non-deterministic behavior C1
C3
C2
Reconfiguration
1. How to test reconfigurations efficiently? 1. How to test reconfigurations efficiently?
2. What is a suitable coverage criteria? 2. What is a suitable coverage criteria?
3. How to ensure liveness of the reconfiguration life cycle? 3. How to ensure liveness of the reconfiguration life cycle?
ES – Real-Time Systems Lab 23 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines
List of Literature
[Cichos et al. 2011] H. Cichos, S. Oster, M. Lochau, A. Schürr : Model-Based Coverage-
Driven Test Suite Generation for Software Product Lines. MoDELS 2011:
425-439
[Cichos et al. 2012] H. Cichos, M. Lochau, S. Oster, A. Schürr : Reduktion von Testsuiten für
Software-Produktlinien. Software Engineering 2012: 143-154
[Lochau et al. 2014] M. Lochau, J. Bürdek, S. Lity, M. Hagner, C. Legat, U. Goltz, A. Schürr :
Applying Model-based Software Product Line Testing Approaches to the
Automation Engineering Domain. AT Journal 2014 (submitted)
[Holzer et al. 2011] A. Holzer, M. Tautschnig, C. Schallhart, H. Veith : An Introduction to Test
Specification in FQL. In S. Barner, I. Harris, D. Kroening, and O. Raz,
editors, Hardware and Software: Verification and Testing, volume 6504 of
Lecture Notes in Computer Science, pages 9–22. Springer Berlin
Heidelberg, 2011.
[Beyer et al. 2011] D. Beyer and M. E. Keremoglu : CPAchecker: A Tool for Configurable
Software Verification. In Proc. CAV, pages 184-190, Springer, 2011.
[Holzmann 1997] G. J. Holzmann : The Model Checker Spin, IEEE Trans. on Software
Engineering, Vol. 23, No. 5, May 1997, pp. 279-295.
ES – Real-Time Systems Lab 24 | 5/8/2014 | Model-based Test Case Generation for (Dynamic) Software Product Lines
List of Literature (2/2)
[Beyer et al. 2004] D. Beyer, A. J. Chlipala, T. A. Henzinger, R. Jhala, and R. Majumdar.
Generating Test from Counterexamples. In Proc. of the 26th ICSE, pages
326–335, 2004.
[Rhein et al. 2011] A. von Rhein, S. Apel, F. Raimondi : Introducing Binary Decision
Diagrams in the Explicit-State Verification of Java Code. Java Pathfinder
Workshop (co-located with ASE'11), 2011.
[Apel et al. 2013] S. Apel, A. von Rhein, P. Wendler, A. Größlinger, D. Beyer : Strategies
for Product-Line Verification: Case Studies and Experiments. In
Proceedings of the IEEE/ACM International Conference on Software
Engineering (ICSE), pages 482–491. IEEE Computer Society, 2013.