Formalization and Comparisonof mcdc and Object Branch

Coverage Criteria

Cyrille Comar, Jerome Guitton, Olivier Hainque, Thomas Quinot

1 / 17

Coverage for Level A

Considering the highest level of certification for aircraft (A inDO-178B):

I Test coverage goal: mcdc

I Structural (source) coverage

I More than statement coverage...I ...but not all execution paths (too costly)I e.g. for a decision C1 and C2 and . . . and CN :

I 2N execution pathsI N + 1 tests in mcdc

2 / 17

Coverage for Level A

Considering the highest level of certification for aircraft (A inDO-178B):

I Test coverage goal: mcdc

I Structural (source) coverage

I More than statement coverage...I ...but not all execution paths (too costly)I e.g. for a decision C1 and C2 and . . . and CN :

I 2N execution pathsI N + 1 tests in mcdc

2 / 17

Coverage for Level A

Considering the highest level of certification for aircraft (A inDO-178B):

I Test coverage goal: mcdc

I Structural (source) coverage

I More than statement coverage...

I ...but not all execution paths (too costly)I e.g. for a decision C1 and C2 and . . . and CN :

I 2N execution pathsI N + 1 tests in mcdc

2 / 17

Coverage for Level A

Considering the highest level of certification for aircraft (A inDO-178B):

I Test coverage goal: mcdc

I Structural (source) coverage

I More than statement coverage...I ...but not all execution paths (too costly)

I e.g. for a decision C1 and C2 and . . . and CN :I 2N execution pathsI N + 1 tests in mcdc

2 / 17

Coverage for Level A

Considering the highest level of certification for aircraft (A inDO-178B):

I Test coverage goal: mcdc

I Structural (source) coverage

I More than statement coverage...I ...but not all execution paths (too costly)I e.g. for a decision C1 and C2 and . . . and CN :

I 2N execution pathsI N + 1 tests in mcdc

2 / 17

Coverage for Level A

Considering the highest level of certification for aircraft (A inDO-178B):

I Test coverage goal: mcdc

I Structural (source) coverage

I More than statement coverage...I ...but not all execution paths (too costly)I e.g. for a decision C1 and C2 and . . . and CN :

I 2N execution paths

I N + 1 tests in mcdc

2 / 17

Coverage for Level A

Considering the highest level of certification for aircraft (A inDO-178B):

I Test coverage goal: mcdc

I Structural (source) coverage

I More than statement coverage...I ...but not all execution paths (too costly)I e.g. for a decision C1 and C2 and . . . and CN :

I 2N execution pathsI N + 1 tests in mcdc

2 / 17

Some issues with mcdc

I source coverage criteria means language-dependent

I no tool for Ada 2005 when Couverture started...

I instrumenting source code? intrusive...

I unbounded execution traces

I Use object coverage instead?

3 / 17

Some issues with mcdc

I source coverage criteria means language-dependent

I no tool for Ada 2005 when Couverture started...

I instrumenting source code? intrusive...

I unbounded execution traces

I Use object coverage instead?

3 / 17

Some issues with mcdc

I source coverage criteria means language-dependent

I no tool for Ada 2005 when Couverture started...

I instrumenting source code? intrusive...

I unbounded execution traces

I Use object coverage instead?

3 / 17

Some issues with mcdc

I source coverage criteria means language-dependent

I no tool for Ada 2005 when Couverture started...

I instrumenting source code? intrusive...

I unbounded execution traces

I Use object coverage instead?

3 / 17

Some issues with mcdc

I source coverage criteria means language-dependent

I no tool for Ada 2005 when Couverture started...

I instrumenting source code? intrusive...

I unbounded execution traces

I Use object coverage instead?

3 / 17

Object coverage to assess mcdc

I Assumption that object branch coverage (obc) is strongerthan mcdc

I widespread industrial practise

I language-independent

I bounded traces

4 / 17

Object coverage to assess mcdc

I Assumption that object branch coverage (obc) is strongerthan mcdc

I widespread industrial practise

I language-independent

I bounded traces

4 / 17

Object coverage to assess mcdc

I Assumption that object branch coverage (obc) is strongerthan mcdc

I widespread industrial practise

I language-independent

I bounded traces

4 / 17

Object coverage to assess mcdc

I Assumption that object branch coverage (obc) is strongerthan mcdc

I widespread industrial practise

I language-independent

I bounded traces

4 / 17

Using obc to achieve mcdc ? short-circuit operators...

(A mod B = 0) and then (C = 0)

A mod B = 0

F

F

C = 0

F

F

T

T

T

B = −1

A > 0

B < 0

F

R = 0

F

F

C = 0

F

F

T

T

T

F

T

F TT

A mod B = 0

5 / 17

Using obc to achieve mcdc ? short-circuit operators...

(A mod B = 0) and then (C = 0)

A mod B = 0

F

F

C = 0

F

F

T

T

T

B = −1

A > 0

B < 0

F

R = 0

F

F

C = 0

F

F

T

T

T

F

T

F TT

A mod B = 0

5 / 17

Using obc to achieve mcdc ? short-circuit operators...

(A mod B = 0) and then (C = 0)

A mod B = 0

F

F

C = 0

F

F

T

T

T

B = −1

A > 0

B < 0

F

R = 0

F

F

C = 0

F

F

T

T

T

F

T

F TT

A mod B = 0

5 / 17

What do certification standards say about that?

DO-248C FAQ #42: Object coverage can be used as long asanalysis can be provided which demonstrates that the coverageanalysis conducted at the Object Code will achieve a comparablelevel of coverage assurance as that conducted at the Source Codelevel.

6 / 17

The sad truth...

I The assumption is wrong: obc is not stronger than mcdc!

I A counterexample in DOT/FAA/AR-07/20, Jun 2007: ObjectOriented Technology Verification Phase 3 Report - StructuralCoverage at the Source Code and Object Code Levels

I (A and then B) or else C can be covered for obc withoutachieving mcdc

7 / 17

The sad truth...

I The assumption is wrong: obc is not stronger than mcdc!

I A counterexample in DOT/FAA/AR-07/20, Jun 2007: ObjectOriented Technology Verification Phase 3 Report - StructuralCoverage at the Source Code and Object Code Levels

I (A and then B) or else C can be covered for obc withoutachieving mcdc

7 / 17

The sad truth...

I The assumption is wrong: obc is not stronger than mcdc!

I A counterexample in DOT/FAA/AR-07/20, Jun 2007: ObjectOriented Technology Verification Phase 3 Report - StructuralCoverage at the Source Code and Object Code Levels

I (A and then B) or else C can be covered for obc withoutachieving mcdc

7 / 17

Having a closer look...

Alloy model: check conjectures, generate counterexamples...

8 / 17

Pathological case

Alloy helped finding a impressive counterexample:

I Decision with an arbitrary high number of conditions N...

I ...that needs N+1 tests to be mcdc-covered...

I ...and only 3 tests to be obc-covered!

Here obc is much weaker than mcdc!

9 / 17

Pathological case

Alloy helped finding a impressive counterexample:

I Decision with an arbitrary high number of conditions N...

I ...that needs N+1 tests to be mcdc-covered...

I ...and only 3 tests to be obc-covered!

Here obc is much weaker than mcdc!

9 / 17

Pathological case

Alloy helped finding a impressive counterexample:

I Decision with an arbitrary high number of conditions N...

I ...that needs N+1 tests to be mcdc-covered...

I ...and only 3 tests to be obc-covered!

Here obc is much weaker than mcdc!

9 / 17

Pathological case

Alloy helped finding a impressive counterexample:

I Decision with an arbitrary high number of conditions N...

I ...that needs N+1 tests to be mcdc-covered...

I ...and only 3 tests to be obc-covered!

Here obc is much weaker than mcdc!

9 / 17

Pathological case

Alloy helped finding a impressive counterexample:

I Decision with an arbitrary high number of conditions N...

I ...that needs N+1 tests to be mcdc-covered...

I ...and only 3 tests to be obc-covered!

Here obc is much weaker than mcdc!

9 / 17

Now what?

I Sure, in some cases, obc does not imply mcdc...

I ...and in some pathological cases, the two criteria divergesquite badly...

I ...but past experience has shown that it works in many cases

I ...so are there conditions that would allow this implication?

I could be enforced by a coding standardI could be an optimization for coverage tools

10 / 17

Now what?

I Sure, in some cases, obc does not imply mcdc...

I ...and in some pathological cases, the two criteria divergesquite badly...

I ...but past experience has shown that it works in many cases

I ...so are there conditions that would allow this implication?

I could be enforced by a coding standardI could be an optimization for coverage tools

10 / 17

Now what?

I Sure, in some cases, obc does not imply mcdc...

I ...and in some pathological cases, the two criteria divergesquite badly...

I ...but past experience has shown that it works in many cases

I ...so are there conditions that would allow this implication?

I could be enforced by a coding standardI could be an optimization for coverage tools

10 / 17

Now what?

I Sure, in some cases, obc does not imply mcdc...

I ...and in some pathological cases, the two criteria divergesquite badly...

I ...but past experience has shown that it works in many cases

I ...so are there conditions that would allow this implication?

I could be enforced by a coding standardI could be an optimization for coverage tools

10 / 17

Now what?

I Sure, in some cases, obc does not imply mcdc...

I ...and in some pathological cases, the two criteria divergesquite badly...

I ...but past experience has shown that it works in many cases

I ...so are there conditions that would allow this implication?

I could be enforced by a coding standard

I could be an optimization for coverage tools

10 / 17

Now what?

I Sure, in some cases, obc does not imply mcdc...

I ...and in some pathological cases, the two criteria divergesquite badly...

I ...but past experience has shown that it works in many cases

I ...so are there conditions that would allow this implication?

I could be enforced by a coding standardI could be an optimization for coverage tools

10 / 17

Cases where obc implies mcdc

Theorem

If there is only one execution path to each condition, then obcimplies mcdc.

C1 and then C2

C1

F

F

C2

F

F

T

T

T

11 / 17

Cases where obc implies mcdc

Theorem

If there is only one execution path to each condition, then obcimplies mcdc.

C1 and then C2

C1

F

F

C2

F

F

T

T

T

11 / 17

Cases where obc implies mcdc

Theorem

If there is only one execution path to each condition, then obcimplies mcdc.

C1 and then C2

C1

F

F

C2

F

F

T

T

T

11 / 17

Cases where obc implies mcdc

Theorem

If there is only one execution path to each condition, then obcimplies mcdc.

C1 and then C2

C1

F

F

C2

F

F

T

T

T

11 / 17

Cases where obc does not imply mcdc

Theorem

On the contrary, if there exists a condition that can be reached bymore than one execution path, obc does not always imply mcdc.

(A and then B) or else C

A

B

T C

T F

12 / 17

Cases where obc does not imply mcdc

Theorem

On the contrary, if there exists a condition that can be reached bymore than one execution path, obc does not always imply mcdc.

(A and then B) or else C

A

B

T C

T F

12 / 17

Cases where obc does not imply mcdc

Theorem

On the contrary, if there exists a condition that can be reached bymore than one execution path, obc does not always imply mcdc.

(A and then B) or else C

A

B

T C

T F

12 / 17

Cases where obc does not imply mcdc

Theorem

On the contrary, if there exists a condition that can be reached bymore than one execution path, obc does not always imply mcdc.

(A and then B) or else C

A

B

T C

T F

12 / 17

human-readable characteristic

Dec1 or else (Dec2 and then . . . )

OR ELSE

Dec

1

AND THEN

Dec

2

[. . .

]

13 / 17

human-readable characteristic

Dec1 or else (Dec2 and then . . . )

OR ELSE

Dec

1

AND THEN

Dec

2

[. . .

]

13 / 17

human-readable characteristic

Dec1 or else (Dec2 and then . . . )

OR ELSE

Dec

1

AND THEN

Dec

2

[. . .

]

no and then

13 / 17

human-readable characteristic

Dec1 or else (Dec2 and then . . . )

OR ELSE

Dec

1

AND THEN

Dec

2

[. . .

]

no or else

13 / 17

Experimental results

In the industrial applications that we looked at, 99 % of thedecisions are such that obc implies mcdc

configuration App. 1 App. 2 GNATcoverage

#decisions 869 37324 1026

#non-tree BDD 7 (0.8 %) 141 (0.4 %) 4 (0.4 %)

14 / 17

Experimental results

In the industrial applications that we looked at, 99 % of thedecisions are such that obc implies mcdc

configuration App. 1 App. 2 GNATcoverage

#decisions 869 37324 1026

#non-tree BDD 7 (0.8 %) 141 (0.4 %) 4 (0.4 %)

14 / 17

Experimental results

In the industrial applications that we looked at, 99 % of thedecisions are such that obc implies mcdc

configuration App. 1 App. 2 GNATcoverage

#decisions 869 37324 1026

#non-tree BDD 7 (0.8 %) 141 (0.4 %) 4 (0.4 %)

14 / 17

Experimental results

I Evaluating the impact of this optimization on the qualificationtestsuite of GNATcoverage;

I compute the coverage of GNATcoverage in 3 differentconfigurations:

I obc: as a baseline;I mcdc 1: historical traces on branches of all decision;I mcdc 2: historical traces only when there are conditions

reachable by several paths...

I ...and compare the size of the generated traces.

15 / 17

Experimental results

I Evaluating the impact of this optimization on the qualificationtestsuite of GNATcoverage;

I compute the coverage of GNATcoverage in 3 differentconfigurations:

I obc: as a baseline;I mcdc 1: historical traces on branches of all decision;I mcdc 2: historical traces only when there are conditions

reachable by several paths...

I ...and compare the size of the generated traces.

15 / 17

Experimental results

I Evaluating the impact of this optimization on the qualificationtestsuite of GNATcoverage;

I compute the coverage of GNATcoverage in 3 differentconfigurations:

I obc: as a baseline;I mcdc 1: historical traces on branches of all decision;I mcdc 2: historical traces only when there are conditions

reachable by several paths...

I ...and compare the size of the generated traces.

15 / 17

Experimental results

configuration obc mcdc 1 mcdc 2

#branches to trace 0 1788 22

size of traces 1.33G 5.06G 1.37G

I The optimization removes 99 % of historical traces

I The overead compared to obc is marginal

16 / 17

Experimental results

configuration obc mcdc 1 mcdc 2

#branches to trace 0 1788 22

size of traces 1.33G 5.06G 1.37G

I The optimization removes 99 % of historical traces

I The overead compared to obc is marginal

16 / 17

Experimental results

configuration obc mcdc 1 mcdc 2

#branches to trace 0 1788 22

size of traces 1.33G 5.06G 1.37G

I The optimization removes 99 % of historical traces

I The overead compared to obc is marginal

16 / 17

More information...

I Resources: Couverture public repository on the Open-Do forge(https://forge.open-do.org/projects/couverture/)

I Alloy modelsI Proofs of theorems comparing obc and mcdcI other results about mcdc

I Couverture project gave birth to an industrial tool:GNATcoverage

17 / 17

More information...

I Resources: Couverture public repository on the Open-Do forge(https://forge.open-do.org/projects/couverture/)

I Alloy models

I Proofs of theorems comparing obc and mcdcI other results about mcdc

I Couverture project gave birth to an industrial tool:GNATcoverage

17 / 17

More information...

I Resources: Couverture public repository on the Open-Do forge(https://forge.open-do.org/projects/couverture/)

I Alloy modelsI Proofs of theorems comparing obc and mcdc

I other results about mcdc

I Couverture project gave birth to an industrial tool:GNATcoverage

17 / 17

More information...

I Resources: Couverture public repository on the Open-Do forge(https://forge.open-do.org/projects/couverture/)

I Alloy modelsI Proofs of theorems comparing obc and mcdcI other results about mcdc

I Couverture project gave birth to an industrial tool:GNATcoverage

17 / 17

More information...

I Resources: Couverture public repository on the Open-Do forge(https://forge.open-do.org/projects/couverture/)

I Alloy modelsI Proofs of theorems comparing obc and mcdcI other results about mcdc

I Couverture project gave birth to an industrial tool:GNATcoverage

17 / 17