Embed Size (px)
DESCRIPTION
Software code review, i.e., the practice of having third-party team members critique changes to a software system, is a well-established best practice in both open source and proprietary software domains. Prior work has shown that the formal code inspections of the past tend to improve the quality of software delivered by students and small teams. However, the formal code inspection process mandates strict review criteria (e.g., in-person meetings and reviewer checklists) to ensure a base level of review quality, while the modern, lightweight code reviewing process does not. Although recent work explores the modern code review process qualitatively, little research quantitatively explores the relationship between properties of the modern code review process and software quality. Hence, in this paper, we study the relationship between software quality and: (1) code review coverage, i.e., the proportion of changes that have been code reviewed, and (2) code review participation, i.e., the degree of reviewer involvement in the code review process. Through a case study of the Qt, VTK, and ITK projects, we find that both code review coverage and participation share a significant link with software quality. Low code review coverage and participation are estimated to produce components with up to two and five additional post-release defects respectively. Our results empirically confirm the intuition that poorly reviewed code has a negative impact on software quality in large systems using modern reviewing tools.
Citation preview
The Impact of Code Review Coverage and Participation
on Software Quality
Shane McIntosh
Ahmed E. Hassan
Bram Adams
Yasutaka Kamei
Patch
+ + - -Patch
+ + - -
2
The modern, tool-supported code review process
GerritShane
Bram
Yasu
Patch
+ + - -
Patch
+ + - -
2
The modern, tool-supported code review process
GerritShane
Bram
Yasu
Patch
+ + - -
Patch
+ + - -
2
The modern, tool-supported code review process
Gerrit
Request review
Shane
Bram
Yasu
Patch
+ + - -
Patch
+ + - -
2
The modern, tool-supported code review process
Gerrit
Watch about NULL values at line 3!
Shane
Bram
Yasu
Patch
+ + - -
Patch
+ + - -
2
The modern, tool-supported code review process
Gerrit
Your code sucks!
Watch about NULL values at line 3!
Shane
Bram
Yasu
Patch
+ + - - Patch
+ + - -
2
The modern, tool-supported code review process
Gerrit
Your code sucks!
Watch about NULL values at line 3!
Shane
Bram
Yasu
The modern, tool-supported code review process
3
GerritShane
Bram
Yasu
The modern, tool-supported code review process
3
GerritShane
Bram
Yasu
The modern, tool-supported code review process
3
Upstream
GerritShane
Bram
Yasu
The modern, tool-supported code review process
3
Upstream
GerritShane
Bram
Yasu
Changes are automatically integrated after review
criteria are satisfied
Traditional!Code Inspections
Mandated reviewer checklists and in-person
meetings
Traditional!Code Inspections
Mandated reviewer checklists and in-person
meetings
Measurably!improves!software!quality
Traditional!Code Inspections
Mandated reviewer checklists and in-person
meetings
Modern!Code Review
Lightweight, tool-supported
Measurably!improves!software!quality
Traditional!Code Inspections
Mandated reviewer checklists and in-person
meetings
Modern!Code Review
Lightweight, tool-supported
Measurably!improves!software!quality
???
Do lax reviewing practices have an impact on software quality?
Expert code!reviewer
6
Upstream
Development
Feature developmentDefect repairing
Commit types
Reviews can be linked to commits in version control systems
6
Upstream
Development
Feature developmentDefect repairing
Commit types
Reviews can be linked to commits in version control systems
7
commit b5b46f398c1a04ab443bf22ec08c5efbf72c161e Author: Alan Alpert <[email protected]> Date: Thu Mar 1 20:09:52 2012 +1000 ! Add a hidden XandYAxis enum value For Qt 5 XandYAxis is being renamed to XAndYAxis to more consistently follow capitalization rules. Add an undocumented XandYAxis variable to ease porting. Change-Id: Id9e41dd5578373f5f557937da889a9326ff12e53 Reviewed-by: Martin Jones <[email protected]>
Gerrit
Upstream
Reviews can be linked to commits in version control systems
7
commit b5b46f398c1a04ab443bf22ec08c5efbf72c161e Author: Alan Alpert <[email protected]> Date: Thu Mar 1 20:09:52 2012 +1000 ! Add a hidden XandYAxis enum value For Qt 5 XandYAxis is being renamed to XAndYAxis to more consistently follow capitalization rules. Add an undocumented XandYAxis variable to ease porting. Change-Id: Id9e41dd5578373f5f557937da889a9326ff12e53 Reviewed-by: Martin Jones <[email protected]>
Gerrit
Upstream
Reviews can be linked to commits in version control systems
7
commit b5b46f398c1a04ab443bf22ec08c5efbf72c161e Author: Alan Alpert <[email protected]> Date: Thu Mar 1 20:09:52 2012 +1000 ! Add a hidden XandYAxis enum value For Qt 5 XandYAxis is being renamed to XAndYAxis to more consistently follow capitalization rules. Add an undocumented XandYAxis variable to ease porting. Change-Id: Id9e41dd5578373f5f557937da889a9326ff12e53 Reviewed-by: Martin Jones <[email protected]>
Gerrit
Upstream
Reviews can be linked to commits in version control systems
7
commit b5b46f398c1a04ab443bf22ec08c5efbf72c161e Author: Alan Alpert <[email protected]> Date: Thu Mar 1 20:09:52 2012 +1000 ! Add a hidden XandYAxis enum value For Qt 5 XandYAxis is being renamed to XAndYAxis to more consistently follow capitalization rules. Add an undocumented XandYAxis variable to ease porting. Change-Id: Id9e41dd5578373f5f557937da889a9326ff12e53 Reviewed-by: Martin Jones <[email protected]>
Gerrit
Upstream
Reviews can be linked to commits in version control systems
8
The Baseline
8
Product Metrics
The Baseline
8
Product Metrics
e.g., SLOC, complexity
The Baseline
8
Product Metrics
Process Metrics e.g., SLOC, complexity
The Baseline
8
Product Metrics
Process Metrics e.g., SLOC, complexity
e.g., Prior defects, churn
The Baseline
8
Product Metrics
Process Metrics
Human Factor Metrics
e.g., SLOC, complexity
e.g., Prior defects, churn
The Baseline
8
Product Metrics
Process Metrics
Human Factor Metrics
e.g., SLOC, complexity
e.g., Prior defects, churn
e.g., # Minor contributors
The Baseline
9
The Context
The Context
1,339 components254 defect-prone
v5.0.0
218 components24 defect-prone
v4.3.0170 components15 defect-prone
v5.10.0
1,337 components187 defect-prone
v5.1.0
10
ParticipationCoverage
Do lax reviewing practices have an impact on software quality?
ParticipationCoverage
Do lax reviewing practices have an impact on software quality?
13
Measuring review coverage using the proportion of reviewed changes
13
Upstream Development
13
Measuring review coverage using the proportion of reviewed changes
13
Upstream Development
Gerrit
13
Measuring review coverage using the proportion of reviewed changes
13
Upstream Development
Gerrit
7 out of 10 changes are!linked to code reviews
14
Concrete Predicted Defect Count
Defect Model
Create artificial component setting explanatory variables to median values
Artificial Comp.
500SLOC
Explanatory!Variable Value
Reviewed changes 100%
Churn 150
15
Defect Model
Calculate number of predicted defects
Artificial Comp.
500SLOC
Explanatory!Variable Value
Reviewed changes
Churn 150
Reviewed changes
Pred
icte
d D
efec
t Cou
nt
Concrete Predicted Defect Count
100%
15
Defect Model
Calculate number of predicted defects
Artificial Comp.
500SLOC
Explanatory!Variable Value
Reviewed changes
Churn 150
Reviewed changes
0
Pred
icte
d D
efec
t Cou
nt
Concrete Predicted Defect Count
100%
15
Defect Model
Calculate number of predicted defects
Artificial Comp.
500SLOC
Explanatory!Variable Value
Reviewed changes
Churn 150
Reviewed changes
0
Pred
icte
d D
efec
t Cou
nt
Concrete Predicted Defect Count
100%
16
Defect Model
Artificial Comp.
500SLOC
Explanatory!Variable Value
Reviewed changes 90%
Churn 150
Modify variable under test while keeping other explanatory variables constant
Reviewed changes
Pred
icte
d D
efec
t Cou
nt
Concrete Predicted Defect Count
16
Defect Model
Artificial Comp.
500SLOC
Explanatory!Variable Value
Reviewed changes 90%
Churn 150
Modify variable under test while keeping other explanatory variables constant
Reviewed changes
Pred
icte
d D
efec
t Cou
nt
Concrete Predicted Defect Count
16
Defect Model
Artificial Comp.
500SLOC
Explanatory!Variable Value
Reviewed changes 90%
Churn 150
1
Modify variable under test while keeping other explanatory variables constant
Reviewed changes
Pred
icte
d D
efec
t Cou
nt
Concrete Predicted Defect Count
16
Defect Model
Artificial Comp.
500SLOC
Explanatory!Variable Value
Reviewed changes 90%
Churn 150
1
Modify variable under test while keeping other explanatory variables constant
Reviewed changes
Pred
icte
d D
efec
t Cou
nt
Concrete Predicted Defect Count
● ●
● ● ● ● ● ● ● ●
● ● ● ● ● ● ●0
1
2
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0Proportion of Reviewed Changes
Con
cret
e Pr
edic
ted
Def
ect C
ount
● Qt 5.0.0VTK 5.10.0
17
Review coverage doesn’t tell the whole story
● ●
● ● ● ● ● ● ● ●
● ● ● ● ● ● ●0
1
2
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0Proportion of Reviewed Changes
Con
cret
e Pr
edic
ted
Def
ect C
ount
● Qt 5.0.0VTK 5.10.0
17
Review coverage doesn’t tell the whole story
Not a significant factor for Qt 5.1.0 or ITK
ParticipationCoverage
Do lax reviewing practices have an impact on software quality?
ParticipationCoverage
Do lax reviewing practices have an impact on software quality?
!Coverage is linked to
quality in two of the four studied systems, yet it doesn’t tell the whole
story
ParticipationCoverage
Do lax reviewing practices have an impact on software quality?
!Coverage is linked to
quality in two of the four studied systems, yet it doesn’t tell the whole
story
20
Measuring review participation using proportions of lax reviews
I❤️ Me
Self-approved changes
20
Measuring review participation using proportions of lax reviews
I❤️ Me
Self-approved changes
Hastily-reviewed changes
20
Measuring review participation using proportions of lax reviews
I❤️ Me
Self-approved changes
Hastily-reviewed changes
Changes without discussion
21
Measuring review participation using proportions of lax reviews
I❤️ Me
Self-approved changes
Hastily-reviewed changes
Changes without discussion
● ● ●
● ●
● ● ●
● ● ●
● ● ●
● ● ●
0
1
2
3
4
5
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0Proportion of Self−Approved Changes
Con
cret
e Pr
edic
ted
Def
ect C
ount
22
Lax participation leads to many post-release defects in our models
● ● ●
● ●
● ● ●
● ● ●
● ● ●
● ● ●
0
1
2
3
4
5
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0Proportion of Self−Approved Changes
Con
cret
e Pr
edic
ted
Def
ect C
ount
22
5 additional post-release defects
Lax participation leads to many post-release defects in our models
● ● ●
● ●
● ● ●
● ● ●
● ● ●
● ● ●
0
1
2
3
4
5
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0Proportion of Self−Approved Changes
Con
cret
e Pr
edic
ted
Def
ect C
ount
22
5 additional post-release defects
Small increase leads to defects
Lax participation leads to many post-release defects in our models
ParticipationCoverage
Do lax reviewing practices have an impact on software quality?
!Coverage is linked to
quality in two of the four studied systems, yet it doesn’t tell the whole
story
ParticipationCoverage
Do lax reviewing practices have an impact on software quality?
!Coverage is linked to
quality in two of the four studied systems, yet it doesn’t tell the whole
story
!Components w/ lax
review participation are estimated to have several additional post-release
defects
![Traffic diversity and code coverage: a preliminary analysisspeed.cis.nctu.edu.tw/~ydlin/pdf/traffic.pdf · Code coverage [11, 12] is a metric defined to present the coverage of](https://img.pdfslide.us/doc/110x75/5f4b0095306fd41a4754ed1c/trafic-diversity-and-code-coverage-a-preliminary-ydlinpdftrafficpdf-code.jpg)
