Rainer Schaden June 30, 2016 · 2016-07-13 · What is Refactoring? É Opdyke - PhD thesis (1992) É Fowler - Refactoring: Improving the Design of Existing Code (1999) Deﬁnition

Threats to the Validity of Refactoring Studies

Rainer SchadenFreie Universität Berlin

June 30, 2016

Outline

Introduction

How to find Refactorings?

Repository Mining Tools

Planned Approach

,

FU Berlin, Threats to the Validity of Refactoring Studies, June 30, 2016 2

Outline

Introduction



Planned Approach

,


What is Refactoring?

É Opdyke - PhD thesis (1992)

É Fowler - Refactoring: Improving the Design of Existing Code (1999)

Definition“A change made to the internal structure of software to make it easier tounderstand and cheaper to modify without changing its observablebehavior”

— Martin Fowler

,



É Opdyke - PhD thesis (1992)É Fowler - Refactoring: Improving the Design of Existing Code (1999)


— Martin Fowler

,



É Opdyke - PhD thesis (1992)É Fowler - Refactoring: Improving the Design of Existing Code (1999)


— Martin Fowler

,


Why Refactoring?

É Prevents software aging

É Improves:É DesignÉ UnderstandabilityÉ MaintainabilityÉ Developer’s productivity

,


Why Refactoring?

É Prevents software agingÉ Improves:

É DesignÉ UnderstandabilityÉ MaintainabilityÉ Developer’s productivity

,


Refactoring Tactics

É Floss Refactoring:É FrequentÉ Mixed with other program changes

É Root-Canal Refactoring:É InfrequentÉ LengthyÉ Hardly any other program changes

,


Refactoring Tactics

É Floss Refactoring:É FrequentÉ Mixed with other program changes

É Root-Canal Refactoring:É InfrequentÉ LengthyÉ Hardly any other program changes

,


Goal of this Thesis

É Examine Refactoring studies with regards to:

É CredibilityÉ RelevanceÉ Validity

,


Goal of this Thesis

É Examine Refactoring studies with regards to:É Credibility

É RelevanceÉ Validity

,


Goal of this Thesis

É Examine Refactoring studies with regards to:É CredibilityÉ Relevance

É Validity

,


Goal of this Thesis

É Examine Refactoring studies with regards to:É CredibilityÉ RelevanceÉ Validity

,


Outline

Introduction



Planned Approach

,



É Goal: empirical refactoring data

É Four Methods: (Murphy-Hill)É Mining the Commit LogÉ Analyzing Code HistoriesÉ Observing ProgrammersÉ Logging Refactoring Tool Use

,



É Goal: empirical refactoring dataÉ Four Methods: (Murphy-Hill)

É Mining the Commit LogÉ Analyzing Code HistoriesÉ Observing ProgrammersÉ Logging Refactoring Tool Use

,



É Goal: empirical refactoring dataÉ Four Methods: (Murphy-Hill)

É Mining the Commit LogÉ Analyzing Code HistoriesÉ Observing ProgrammersÉ Logging Refactoring Tool Use

,


Mining the Commit Log

É Search for "refactor", "rename", etc. in commit messages

É Problems:É Bad commit messagesÉ Floss refactoringÉ Tendency towards certain, large refactorings

É Assumptions:É Developer recalls refactoringÉ And describes it accurately

,



É Search for "refactor", "rename", etc. in commit messagesÉ Problems:

É Bad commit messagesÉ Floss refactoringÉ Tendency towards certain, large refactorings


,



É Search for "refactor", "rename", etc. in commit messagesÉ Problems:

É Bad commit messagesÉ Floss refactoringÉ Tendency towards certain, large refactorings


,


Analyzing Code Histories

É Analyzing versions of source code

É Problems:É Manual inspection is slow and error-proneÉ Comparing non-consecutive versionsÉ Tools only recognize limited number of refactoringsÉ Tools use heuristics

É Assumptions:É Adequate granularity in code historyÉ Appropriate window of observationÉ Heuristics are parameterized correctly

,



É Analyzing versions of source codeÉ Problems:

É Manual inspection is slow and error-proneÉ Comparing non-consecutive versionsÉ Tools only recognize limited number of refactoringsÉ Tools use heuristics


,



É Analyzing versions of source codeÉ Problems:

É Manual inspection is slow and error-proneÉ Comparing non-consecutive versionsÉ Tools only recognize limited number of refactoringsÉ Tools use heuristics


,


Observing Programmers

É Direct observation: controlled experiment

É Indirect observation: surveyÉ Problems:

É Controlled experiments are expensiveÉ Bad external validity of controlled experimentsÉ Indirect observation relies on memory of developers

É Assumptions:É Developers recall refactorings accuratelyÉ Frequent refactoringsÉ External validity

,



É Direct observation: controlled experimentÉ Indirect observation: survey

É Problems:É Controlled experiments are expensiveÉ Bad external validity of controlled experimentsÉ Indirect observation relies on memory of developers


,



É Direct observation: controlled experimentÉ Indirect observation: surveyÉ Problems:



,



É Direct observation: controlled experimentÉ Indirect observation: surveyÉ Problems:



,


Logging Refactoring Tool Use

É Use log files of automated refactoring tools

É Problems:É Requires extensive log filesÉ Can’t track manual refactorings

É Assumptions:É Developers use tools for refactoring

,



É Use log files of automated refactoring toolsÉ Problems:

É Requires extensive log filesÉ Can’t track manual refactorings


,



É Use log files of automated refactoring toolsÉ Problems:

É Requires extensive log filesÉ Can’t track manual refactorings


,


Strength and Weaknesses

É Implicit/ExplicitÉ Accuracy and PrecisionÉ ContextÉ Fidelity

,


Outline

Introduction



Planned Approach

,


Ref-Finder

É Prete et al. (2010)É Analyzing versions of source codeÉ Supports 63 refactoringsÉ Logic rules

,


Ref-Finder Description

É Input: Two versions of a program

É Basic factsÉ I.e. method(methodFullName, methodShortName, typeFullName)

É Deleted_ und added_ factsÉ Similarbody factsÉ Refactorings are rulesÉ Topological sort

,



É Input: Two versions of a programÉ Basic facts

É I.e. method(methodFullName, methodShortName, typeFullName)


,





É Deleted_ und added_ facts

É Similarbody factsÉ Refactorings are rulesÉ Topological sort

,





É Deleted_ und added_ factsÉ Similarbody facts

É Refactorings are rulesÉ Topological sort

,





É Deleted_ und added_ factsÉ Similarbody factsÉ Refactorings are rules

É Topological sort

,






,


Ref-Finder Example Rule

Example

É deleted_method(mFullName, mShortName, t1FullName) ∧

É added_method(newmFullName, mShortName, t2FullName) ∧É similar_body(newmFullName, newmBody, mFullName, mBody) ∧É NOT(equals(t1FullName, t2FullName))É → move_method(mShortName, t1FullName, t2FullName)

,



Example

É deleted_method(mFullName, mShortName, t1FullName) ∧É added_method(newmFullName, mShortName, t2FullName) ∧

É similar_body(newmFullName, newmBody, mFullName, mBody) ∧É NOT(equals(t1FullName, t2FullName))É → move_method(mShortName, t1FullName, t2FullName)

,



Example

É deleted_method(mFullName, mShortName, t1FullName) ∧É added_method(newmFullName, mShortName, t2FullName) ∧É similar_body(newmFullName, newmBody, mFullName, mBody) ∧

É NOT(equals(t1FullName, t2FullName))É → move_method(mShortName, t1FullName, t2FullName)

,



Example

É deleted_method(mFullName, mShortName, t1FullName) ∧É added_method(newmFullName, mShortName, t2FullName) ∧É similar_body(newmFullName, newmBody, mFullName, mBody) ∧É NOT(equals(t1FullName, t2FullName))

É → move_method(mShortName, t1FullName, t2FullName)

,



Example

É deleted_method(mFullName, mShortName, t1FullName) ∧É added_method(newmFullName, mShortName, t2FullName) ∧É similar_body(newmFullName, newmBody, mFullName, mBody) ∧É NOT(equals(t1FullName, t2FullName))É → move_method(mShortName, t1FullName, t2FullName)

,


Ref-Finder Problems: Missing Refactorings

É Nine of 72 refactorings are not detectedÉ Including “Substitute Algorithm”

,


Ref-Finder Problems: Wrong precision

É Precision =|E ∩ R||R|

É Recall = |E ∩ R||E|

Figure : Ref-Finder precision(Source: Prete et al.)

,






,






,






,


Ref-Finder Problems: Questionable Recall

Recall:“Since it is hard to find known refactorings, we ran REF-FINDER usingsimilarity threshold σ = 0.65 and manually inspected randomly chosenrefactorings until we found 10 correct refactorings. We then measured arecall against this data set at a more reasonable threshold, σ = 0.85.”

— Prete et al.

,


UMLDiff

É Xing and Stroulia (2005)É Structural-differencing algorithmÉ Name-similarity and Structure-similarity heuristicsÉ Supports 33 Refactorings

,


RefactoringCrawler

É Dig et al. (2006)É Syntactic analysis using Shingles encodingÉ Semantic analysisÉ Supports seven Refactorings

,


Outline

Introduction



Planned Approach

,


Planned Approach

É Examine precision and recall of Ref-Finder

É Analyze results of studies using Ref-FinderÉ Examine precision and recall of other tools and methodsÉ Analyze results of other studies

,


Planned Approach

É Examine precision and recall of Ref-FinderÉ Analyze results of studies using Ref-Finder

É Examine precision and recall of other tools and methodsÉ Analyze results of other studies

,


Planned Approach

É Examine precision and recall of Ref-FinderÉ Analyze results of studies using Ref-FinderÉ Examine precision and recall of other tools and methods

É Analyze results of other studies

,


Planned Approach

É Examine precision and recall of Ref-FinderÉ Analyze results of studies using Ref-FinderÉ Examine precision and recall of other tools and methodsÉ Analyze results of other studies

,


Further possibilities: Time pressure andreleases

Recommendation“The other time you should avoid refactoring is when you are close to adeadline. At that point the productivity gain from refactoring would onlyappear after the deadline and thus be too late.”

— Martin Fowler

É Studies equate time pressure with major releasesÉ Questionable for Open-source software

,


Further possibilities: Time pressure andreleases

Recommendation“The other time you should avoid refactoring is when you are close to adeadline. At that point the productivity gain from refactoring would onlyappear after the deadline and thus be too late.”

— Martin Fowler

É Studies equate time pressure with major releasesÉ Questionable for Open-source software

,


Further possibilities: Bug-inducing vs.Bug-fix-inducing changes

É Are refactorings really responsible for bugs?É Or do they help finding bugs?

,


Summary

É There are four basic methods to gather empirical data aboutrefactoring

É Tools to detect refactorings are necessary but flawedÉ Goal: How valid are the results of refactoring studies?

,


For Further Reading I

Martin FowlerRefactoring: Improving the Design of Existing Code.Addison-Wesley Longman Publishing Co., Inc., 1999.

Murphy-Hill, Black, Dig, ParninGathering Refactoring Data: A Comparison of Four MethodsProceedings of the 2nd Workshop on Refactoring Tools, 7:1–7:5, 2008.

Prete, Rachatasumrit, Sudan, KimTemplate-based Reconstruction of Complex RefactoringsProceedings of the 2010 IEEE International Conference on SoftwareMaintenance, 1–10, 2010.

,


Thank You!

,


Documents

Rainer Schaden June 30, 2016 · 2016-07-13 · What is Refactoring? É Opdyke - PhD thesis (1992) É Fowler - Refactoring: Improving the Design of Existing Code (1999) Deﬁnition