Improving Coverage Analysis - zenspider.com · 2020-05-16 · Improving Coverage Analysis RubyConf 2016, Cincinnati, OH Ryan Davis, Seattle.rb @the_zenspider Ryan Davis Coding professionally

Improving Coverage Analysis

RubyConf 2016, Cincinnati, OH

Ryan Davis, Seattle.rb

@the_zenspider


1




@the_zenspider

Ryan DavisCoding professionally 25 years, 16 years RubyFounder, Seattle.rb; First & Oldest Ruby BrigadeAuthor: minitest, flog, flay, debride, ruby_parser, etc.~114 million downloaded gems (114/10648 = 1%!)Developer’s Developer: I ❤ building toolsRun Seattle.rb ConsultingMost text of any slide

2




@the_zenspider

Setting ExpectationsConceptual / Idea TalkBeginner+144 slides = 3-5 spm (easy pace)Shooting for ~30 minutes

3




@the_zenspider

Introduction

4




@the_zenspider

As a Consultant

5




@the_zenspider

I get clients

6




@the_zenspider

Huge/Messy Implementations

7




@the_zenspider

Tiny/No Tests

8




@the_zenspider

Making working on it

dangerous/impossible9




@the_zenspider

Pair 100%?

10




@the_zenspider

(Not Realistic)

11




@the_zenspider

PRs in Limbo

12




@the_zenspider

for Months🗓

13




@the_zenspider

It’s Frustrating 😤😡💥🌋

14




@the_zenspider

To Be a Fixer

15




@the_zenspider

But Not Fix Things

16




@the_zenspider

Being a Tool Builder

17




@the_zenspider

Lots of tools

18




@the_zenspider

To Help Me

19




@the_zenspider

Flog

20




@the_zenspider

Flay

21




@the_zenspider

Debride

22




@the_zenspider

Minitest

23




@the_zenspider

Minitest-bisect

24




@the_zenspider

But…

25




@the_zenspider

Without Tests?

26




@the_zenspider

I’m not getting very far

27




@the_zenspider

How to Improve Tests?

28




@the_zenspider

Code Coverage

29




@the_zenspider

What is Code Coverage Analysis?

30




@the_zenspider

Miller & Maloney Communications of

the ACM31




@the_zenspider

in 1963!!!

32




@the_zenspider

Everything good in computing

33




@the_zenspider

is old.

34




@the_zenspider

Like me.

35




@the_zenspider

The Measure of the Amount of the Code that

the Tests execute36




@the_zenspider

class Adder def initialize n @n = n end

def add m @n + m end end

100% Covered!

Huh?

37

class TestAdder < Minitest::Test def test_add assert_equal 3, Adder.new(1).add(2) end end




@the_zenspider

Lots of Types

38




@the_zenspider

C0-C2C0 Statement/Line coverageC1 Branch coverageC2 Condition coverage

39

http://atlas-softquality.com/faqanswer3.html




@the_zenspider

Function Coverage

40

✅ MyClass#method1

◻ MyClass#method2

✅ MyClass#method3




@the_zenspider

Statement executed?

(C0)41




@the_zenspider

if condition then # truthy else # falsey end

Branches exhausted?

(C1)42




@the_zenspider

if (a or b) and c then # truthy else # falsey end

Conditions exhausted?

(C2)43




@the_zenspider

(a or b) and c

1 1 1 -> 10 0 0 -> 0

0 0 1 -> 01 0 1 -> 10 1 1 -> 10 1 0 -> 0

0 0 0 -> 00 0 1 -> 00 1 0 -> 00 1 1 -> 11 0 0 -> 01 0 1 -> 11 1 0 -> 01 1 1 -> 1

DecisionCoverage

decision

truthy falsey

44

ModifiedDecision/ConditionCoverage

and

or

FullConditionCoverage

c

and

or

a b




@the_zenspider

Parameter Edge Case Coverage

45

Strings:

NullEmpty

WhitespaceValid Format

Invalid FormatSingle-ByteMulti-Byte




@the_zenspider

Path Coverage

46

1

2

3

4

5

6




@the_zenspider

Entry/Exit Coveragedef blah return unless blah?

if blah then x = y return z end

x end

47




@the_zenspider

State Coverage

48

def initialize str @str = str end




@the_zenspider49

Warning!




@the_zenspider

MetricsAhead!

50

Bad




@the_zenspider

This is NOT a Dilbert Cartoon

51




@the_zenspider

Because Fuck Scott Adams

52

http://rationalwiki.org/wiki/Scott_Adams




@the_zenspider

Soooo…

53




@the_zenspider

People Get Caught Up

54




@the_zenspider

In Getting High Scores

55




@the_zenspider

Leading to Gaming the System

56




@the_zenspider

But High Code Coverage

57




@the_zenspider

IS NOT

58




@the_zenspider

Proof of Good Tests

59




@the_zenspider

“But we have 110% code coverage, we can't have

bugs” —Actual Quote

60




@the_zenspider

Proofclass Adder def initialize n @n = n end

def add m @n + m end end

61




@the_zenspider

Proofclass TestAdder < Minitest::Test def test_add assert_equal 3, Adder.new(1).add(2) end end

62




@the_zenspider

Proofclass TestAdder < Minitest::Test def test_add assert_equal 3, Adder.new(1).add(2) end end

63




@the_zenspider

100% Covered!

Still 100%

Covered!?!

Proofclass TestAdder < Minitest::Test def test_add Adder.new(1).add(2) end end

64




@the_zenspider

TDD65

Tothe

Rescue!




@the_zenspider

Red Green

Refactor66




@the_zenspider

Natural Fix

67




@the_zenspider

Many other Benefits

68




@the_zenspider

Current State of the Art

69




@the_zenspider

Coverage

70




@the_zenspider

Nobody Knows About It

71




@the_zenspider

Because…

72




@the_zenspider

Ships w/ Ruby

73




@the_zenspider

Coveragerequire "coverage" Coverage.start # must be first-ish require "the_code" require "the_tests" Tests.run Coverage.result # => { … }

74




@the_zenspider

Coveragepp Coverage.result

{ "/full/path_to_file.rb" => [ nil, # comment/blank 0, # not covered 1+ # covered N times ], … }

75




@the_zenspider

Not For Human Consumption ☹

76




@the_zenspider

How does it Work?

77




@the_zenspider

Hooks in VM

78




@the_zenspider

Coverage.start

-> sets hash79




@the_zenspider

Load or Eval

80




@the_zenspider

Extra VM Instructionssasm: #<ISeq:<compiled>@<compiled>>================================ +trace 131072 ( 12) putspecialobject 1 ( 12) putobject :foo putiseq foo send <callinfo!mid:core#define_method, argc:2, ARGS_SIMPLE>, <callcache>, nil pop +trace 131072 ( 16) putself ( 16) send <callinfo!mid:foo, argc:0, FCALL|VCALL|ARGS_SIMPLE>, <callcache>, nil putobject 10 send <callinfo!mid:+, argc:1, ARGS_SIMPLE>, <callcache>, nil leave sasm: #<ISeq:foo@<compiled>>======================================= +trace 131072 ( 13) putself ( 13) send <callinfo!mid:bar, argc:0, FCALL|VCALL|ARGS_SIMPLE>, <callcache>, nil leave

81




@the_zenspider

tests run code

82




@the_zenspider

increments counters

83




@the_zenspider

Coverage.result

-> returns & clears hash84




@the_zenspider

Coverage.peek_result

-> returns hash85




@the_zenspider

Simplecov

86




@the_zenspider

Code Examplerequire "simplecov" SimpleCov.start # must be first-ish require "the_code" require "the_tests" Tests.run

87




@the_zenspider

Simplecov

Uses Coverage Internally.Improves Reporting.

88




@the_zenspider89




@the_zenspider90




@the_zenspider

Not Much Else

91




@the_zenspider

The Problem™

92




@the_zenspider

But First…

93




@the_zenspider

Types of Errors

94




@the_zenspider

Statistics

95




@the_zenspider

Type 1: False Positive

96




@the_zenspider

6 + 0 + 8 + 48 + 4 + 8 + 4 = 18

24 = 75%

97

class X

blah blahblahif blah blah blah blahelse blahend

blah blahblahblah blah blahblah

class Y



class TestX

blah blah

assert blah




@the_zenspider

Type 2: False Negative

98




@the_zenspider99

class X



class Y



class TestX

blah blah

assert blah

class TestQ

blah blah

assert blah

???

18 + 024 + n

≤ 75%




@the_zenspider

Numerator Errors

100




@the_zenspider

“Type 3”: Error by Omission

101




@the_zenspider102

class X



class Y



class TestX

blah blah

assert blah

class Z

??? ???????? ??? ??? ??? ?????? ??????

??? ????????? ??? ??????

6 + 0 + 8 + 4 + n8 + 4 + 8 + 4 + m

= ??%




@the_zenspider

Denominator Error

103




@the_zenspider

OK…

104




@the_zenspider

How Coverage

Sucks105




@the_zenspider

Type 1 Errors: Macro Type 1 Errors: Micro Type 2 Errors: None Type 3 Errors: Maybe

106




@the_zenspider

class TestY

blah blah

assert blah

class Y



Never Directly Tested

class X



Type 1 Error: Macro

107

class Z



class TestX

blah blah

assert blah

class TestZ

blah blah

assert blah




@the_zenspider

Type 1Error: Microbuggy_response if false # line coveredraise "haha" unless true # line covered

a ? b : c # line covereda ? b : c # line covereda = b rescue c # line covereda = b || c # line covered

has_many :problems # line covered

a; b; c; d; e; # line coveredi_raise; d; e; # line covered (and so on…)

108




@the_zenspider

No Type 2 Errors in simplecov

109




@the_zenspider

class Z



Type 3 Errors

110

class TestY

blah blah

assert blah

class Y



class X



class TestX

blah blah

assert blah

class TestZ

blah blah

assert blah

Don'tWrite

Don'tRun

NeverLoaded

RailsLazy

Loading




@the_zenspider

How can it be improved?

111




@the_zenspider

minitest-coverage

112




@the_zenspider

Uses Coverage

113




@the_zenspider

Extends C API (optionally?)

Coverage.result=114




@the_zenspider

New Strategy

115




@the_zenspider

Record Baseline =

Fix Error by Omission116




@the_zenspider

Baseline?

117

“a minimum or starting point used for comparisons.”




@the_zenspider

Loading Everything + Running Nothing

= Baseline118




@the_zenspider

Creating a BaselineENV["RAILS_ENV"] = "production"

require "coverage/start" require "./config/environment.rb"

Dir["app/**/*.rb", "lib/**/*.rb"].each do |path| require path end

require "coverage/pruner"

# (or just use `minitest_coverage_baseline`)119




@the_zenspider

Only record coverage for CUT

120




@the_zenspider

Record for CUT

121

class TestY

blah blah

assert blah

class Y



class X



class TestX

blah blah

assert blah

✂




@the_zenspider

Map Test Class

to Implementation

122

class X



class TestX

blah blah

assert blah

Hard to do




@the_zenspider

Coverage is by Path{ "/full/path_to_file.rb" => [ nil, # comment/blank 0, # not covered 1+ # covered N times ], … }

123




@the_zenspider

Classes ≭ Path% ag class.Something app/models/something_else.rb: 1: class Something

124




@the_zenspider

Test Names ≭ Impl NamesValuations::UpdateWeightedExit

≭

ValuationWeightedExitTest

125




@the_zenspider

Test Names ≭ Impl Namesapp/services/valuations/updated_weighted_exit.rb:

module Valuations class UpdateWeightedExit

test/unit/services/valuations/weighted_exit_test.rb:

class ValuationWeightedExitTest

126




@the_zenspider

Biased Towards False Negatives

127




@the_zenspider

Increased Type 2 Errors

128




@the_zenspider

Fixed by Naming

Conventions129




@the_zenspider

simplecov: 83%

130




@the_zenspider

+ Baseline: 51%

131




@the_zenspider

Fixed Type 2s: 62%

132




@the_zenspider

Report OutputCompanyTest::Create: .............. app/models/company.rb: from 0.0% to 28.6% of 228 lines

CompanyTest::Modify: ...... app/models/company.rb: from 28.6% to 45.6% of 228 lines

133




@the_zenspider

Difference in Numbersuncv covr% totl : path

365 13.5% 422 : app/models/report.rb 294 7.8% 319 : app/controllers/transactions_controller.rb 184 7.1% 198 : app/concerns/formula.rb 145 19.4% 180 : app/controllers/workflows_controller.rb 145 14.7% 170 : app/controllers/permissions_controller.rb 143 30.9% 207 : app/helpers/transactions_helper.rb 142 10.7% 159 : app/helpers/reports_helper.rb ...

134




@the_zenspider

What’s left?

135




@the_zenspider

Hook into simplecov’s

report generator

136




@the_zenspider

emacs & vim visualization

/ coloring

137




@the_zenspider

Better error handling: type 2 errors on

name mismatches138




@the_zenspider

Enhancment: Record per-test &

map lines covered back to each test

139

class X



class TestX

blah blah

assert blah




@the_zenspider

Enhancment: then use that

to show heavy overlaps

140

class X



class TestX

blah blah

assert blah




@the_zenspider

Neat Idea: Do this without

filtering to identify places to isolate/

disconnect141

class Y



class X



class TestX

blah blah

assert blah

✂




@the_zenspider

I'd ❤ Help!

142

https://github.com/seattlerb/minitest-coverage

https://github.com/seattlerb/minitest-coverage




@the_zenspider

Need Tools? Need Help?

I’m Available!143




@the_zenspider

Thank You (hire me)

144

Documents

Improving Coverage Analysis - zenspider.com · 2020-05-16 · Improving Coverage Analysis RubyConf 2016, Cincinnati, OH Ryan Davis, Seattle.rb @the_zenspider Ryan Davis Coding professionally