1

Covrig: A Framework for the Analysis of Code, Test, and Coverage Evolution in Real Software

Paul Marinescu, Petr Hosek, Cristian CadarImperial College London

2

Goal

• Answer questions about software evolution– Code quality– Test quality– Development model– Testing improvement opportunities

…using software development historical data

3

Target Audience

• Researchers– Hypothesis validation (e.g. are software patches

poorly tested?)

• Programmers/Project Managers– Assess development quality

4

Software Metrics

• Static–Measured by parsing the software artifacts

• Dynamic–Require running the evolving software–More challenging–Very few studies

5

Example questions

1. Do executable and test code evolve in sync?2. How many patches touch only code/test/none/both?3. What is the distribution of patch sizes?4. How spread out is each patch through the code?5. Is test suite execution deterministic?6. How does the overall coverage evolve?7. What is the distribution of patch coverage across revisions?8. What is the latent patch coverage?9. Are bug fixes better covered than other patches?10. Is the coverage of buggy code less than average?

6

Data mining infrastructure

Empirical case study

7

Covrig Overview

3

2

1

8

Docker Containers

• Lightweight, OS-level virtualization– Guest shares kernel with host– Namespace isolation

• PID• Network• IPC• Filesystem

– Resource limiting

• cgroups + Linux Containers + Docker

9

Docker Containers Features

• Isolation• Consistency• Reproducibility• Easy cloud deployment• Performance

10

Covrig

11

Metric Granularity

Static

Test size Lines

Executable code size Lines

HunksPatch executable size Files

Dynamic

Overall coverageLines

Branches

Patch coverageLines

Branches

Latent patch coverage Lines

Test result FAIL/PASS

12

Challenges

Evolving dependenciesEvolving containersCustom compile flags (-Wno-error)

13

Challenges

Branching development structureConsider only the ‘main’ branch

Alice Bob

r1

r3

m1

r2

r4

r1

r3

r2+r4

14

Challenges

Revisions that fail to compileAccumulate until reaching a compilable revision

r1

r2

r3 r1+r2+r3

15

Data mining infrastructure

Empirical case study

16

Case Study Subjects

App ELOCTests

Period (mo)Lang LOC

Binutils 27,029 DejaGnu 5,186 35

Git 79,760 C/shell 108,464 5

Lighttpd 23,884 Python 2,440 36

Memcached 4,426 C/Perl 4,605 47

Redis 18,203 Tcl 7,589 6

ZeroMQ 7,276 C++ 3,460 17

1500 revisions and 12 years of development in total

17

Patch type

18

Is test suite execution deterministic?

FAIL/PASS determinismBinutils Git Lighttpd Memcached Redis ZeroMQ

NondeterministicRevisions 0 1 1 21 16 32

19

Is test suite execution deterministic?

Coverage determinismBinutils Git Lighttpd Memcached Redis ZeroMQ

NondeterministicLines (median) 0 13 10 8.5 23 27

20

Test Suite Nondeterminism Causes

• Bugs– Race conditions– Hardcoded wall clock timeouts– Incorrect resource consumption expectations

• Random test data• Benign race conditions

21

Are patches properly tested?

Sometimes

22

Patch coverage

23

Patch coverage

0% 0%

0%0%

0%

0%

24

Does covered code contain fewer bugs that not covered code?

Not really

25

Does covered code contain fewer bugs that not covered code?

Patch Coverage (median) Patches Fully Covered

Buggy All Buggy All

Memcached 100% 89% 67% 45%

Redis 94% 0% 47% 25%

ZeroMQ 71% 76% 37% 33%

85 total bugs

26

Conclusions

Dynamic software metrics mining

Case study on 6 systems/1500 revisions/12 years of development

Open source extensible infrastructurehttp://srg.doc.ic.ac.uk/projects/covrig/