Upload
richard-wettel
View
2.941
Download
8
Tags:
Embed Size (px)
DESCRIPTION
The slides for the presentation I gave at ICSE 2011, in Honolulu, Hawaii.
Citation preview
REVEAL @ Faculty of InformaticsUniversity of Lugano
Switzerland
Richard Wettel, Michele LanzaPLEIAD @ DCC
University of ChileChile
Romain Robbes
Software Systems as Cities:A Controlled Experiment
Software Systems as Cities
City Metaphor
VISSOFT 2007
City Metaphor
class building
package district
VISSOFT 2007
City Metaphor
class building
package district
VISSOFT 2007
City Metaphor
class building
package district
nesting level color
VISSOFT 2007
City Metaphor
class building
package district
number of methods (NOM) height
number of attributes (NOA) base size
number of lines of code (LOC) color
nesting level color
VISSOFT 2007
Program ComprehensionArgoUMLArgoUMLLOC 136,325
ICPC 2007
Program Comprehension
skyscraperFacadeMDRImplFacadeMDRImpl
NOA 3NOM 349LOC 3,413
office buildingCPPParserCPPParser
NOA 85NOM 204LOC 9,111
parking lotJavaTokenTypesJavaTokenTypes
NOA 173NOM 0LOC 0
house PropPanelEventPropPanelEvent
NOA 2NOM 3LOC 37
Program Comprehension
ICPC 2007
Design Quality Assessment
ArgoUML classesArgoUML classesArgoUML classes
brain class 8
god class 30
god + brain 6
data class 17
unaffected 1,715SoftVis 2008
disharmony map
System Evolution Analysis
time traveling
WCRE 2008
System Evolution Analysis
ArgoUML8 major releases6 years
time traveling
WCRE 2008
time
System Evolution Analysis
ArgoUML8 major releases6 years
time traveling
WCRE 2008
time
http://codecity.inf.usi.ch
implemented in Smalltalk
ICSE 2008 tool demo
usefulIs it ?
A Controlled Experiment
Design
State of the art?technical report 2010
Design desiderata1 Avoid comparing using a technique against not using it.
2 Involve participants from the industry.
3 Provide a not-so-short tutorial of the experimental tool to the participants.
4 Avoid, whenever possible, giving the tutorial right before the experiment.
5 Use the tutorial to cover both the research behind the approach and the tool.
6 Find a set of relevant tasks.
7 Choose real object systems that are relevant for the tasks.
8 Include more than one object system in the design.
9 Provide the same data to all participants.
10 Limit the amount of time allowed for solving each task.
11 Provide all the details needed to make the experiment replicable.
12 Report results on individual tasks.
13 Include tasks on which the expected result is not always to the advantage of the tool being evaluated.
14 Take into account the possible wide range of experience level of the participants.
Design desiderata1 Avoid comparing using a technique against not using it.
2 Involve participants from the industry.
3 Provide a not-so-short tutorial of the experimental tool to the participants.
4 Avoid, whenever possible, giving the tutorial right before the experiment.
5 Use the tutorial to cover both the research behind the approach and the tool.
6 Find a set of relevant tasks.
7 Choose real object systems that are relevant for the tasks.
8 Include more than one object system in the design.
9 Provide the same data to all participants.
10 Limit the amount of time allowed for solving each task.
11 Provide all the details needed to make the experiment replicable.
12 Report results on individual tasks.
13 Include tasks on which the expected result is not always to the advantage of the tool being evaluated.
14 Take into account the possible wide range of experience level of the participants.
Finding a baseline
1. program comprehension
2. design quality assessment
3. system evolution analysis
Finding a baseline
1. program comprehension
2. design quality assessment
3. system evolution analysis
Finding a baseline
1. program comprehension
2. design quality assessment
3. system evolution analysis
Finding a baseline
1. program comprehension
2. design quality assessment
Tasks
A1 Identity the convention used in the system to organize unit tests.
A2.1&A2.2
What is the spread of term T in the name of the classes, their attributes and methods?
A3 Evaluate the change impact of class C, in terms of intensity and dispersion.
A4.1 Find the three classes with the highest number of methods.
A4.2Find the three classes with the highest average number of lines of code per method.
Tasksprogram comprehension 6
A1 Identity the convention used in the system to organize unit tests.
A2.1&A2.2
What is the spread of term T in the name of the classes, their attributes and methods?
A3 Evaluate the change impact of class C, in terms of intensity and dispersion.
A4.1 Find the three classes with the highest number of methods.
A4.2Find the three classes with the highest average number of lines of code per method.
Tasksprogram comprehension
design quality assessment 4
6
B1.1 Identify the package with the highest percentage of god classes.
B1.2 Identify the god class with the largest number of methods.
B2.1Identify the dominant (affecting the highest number of classes) class-level design problem.
B2.2 Write an overview of the class-level design problems in the system.
A1 Identity the convention used in the system to organize unit tests.
A2.1&A2.2
What is the spread of term T in the name of the classes, their attributes and methods?
A3 Evaluate the change impact of class C, in terms of intensity and dispersion.
A4.1 Find the three classes with the highest number of methods.
A4.2Find the three classes with the highest average number of lines of code per method.
Tasksprogram comprehension
design quality assessment 4
6
B1.1 Identify the package with the highest percentage of god classes.
B1.2 Identify the god class with the largest number of methods.
B2.1Identify the dominant (affecting the highest number of classes) class-level design problem.
B2.2 Write an overview of the class-level design problems in the system.
5
TasksA1 Identity the convention used in the system to organize unit tests.
A2.1&A2.2
What is the spread of term T in the name of the classes, their attributes and methods?
A3 Evaluate the change impact of class C, in terms of intensity and dispersion.
A4.1 Find the three classes with the highest number of methods.
A4.2Find the three classes with the highest average number of lines of code per method.
B1.1 Identify the package with the highest percentage of god classes.
B1.2 Identify the god class with the largest number of methods.
B2.1Identify the dominant (affecting the highest number of classes) class-level design problem.
B2.2 Write an overview of the class-level design problems in the system.
quantitative
qualitative 1
98
Main research questions
Main research questions
Does the use of CodeCity increase the correctness of the solutions to program comprehension tasks, compared to non-visual exploration tools, regardless of the object system size?1
Main research questions
Does the use of CodeCity increase the correctness of the solutions to program comprehension tasks, compared to non-visual exploration tools, regardless of the object system size?
Does the use of CodeCity reduce the time needed to solve program comprehension tasks, compared to non-visual exploration tools, regardless of the object system size?2
1
Variables of the experiment
Variables of the experiment
dependentcorrectnesscorrectnesscorrectnesscorrectnesscorrectness
dependentcompletion timecompletion timecompletion timecompletion timecompletion time
independenttool CodeCityCodeCityCodeCityCodeCity
independenttool Eclipse + ExcelEclipse + ExcelEclipse + ExcelEclipse + Excelindependentobject system sizeobject system sizeobject system sizeobject system size mediumindependentobject system sizeobject system sizeobject system sizeobject system size large
controlledexperience levelexperience levelexperience level beginnerbeginner
controlledexperience levelexperience levelexperience level advancedadvancedcontrolledbackgroundbackground academiaacademiaacademiacontrolledbackgroundbackground industryindustryindustry
Variables of the experiment
dependentcorrectnesscorrectnesscorrectnesscorrectnesscorrectness
dependentcompletion timecompletion timecompletion timecompletion timecompletion time
independenttool CodeCityCodeCityCodeCityCodeCity
independenttool Eclipse + ExcelEclipse + ExcelEclipse + ExcelEclipse + Excelindependentobject system sizeobject system sizeobject system sizeobject system size mediumindependentobject system sizeobject system sizeobject system sizeobject system size large
controlledexperience levelexperience levelexperience level beginnerbeginner
controlledexperience levelexperience levelexperience level advancedadvancedcontrolledbackgroundbackground academiaacademiaacademiacontrolledbackgroundbackground industryindustryindustry
Variables of the experiment
dependentcorrectnesscorrectnesscorrectnesscorrectnesscorrectness
dependentcompletion timecompletion timecompletion timecompletion timecompletion time
independenttool CodeCityCodeCityCodeCityCodeCity
independenttool Eclipse + ExcelEclipse + ExcelEclipse + ExcelEclipse + Excelindependentobject system sizeobject system sizeobject system sizeobject system size mediumindependentobject system sizeobject system sizeobject system sizeobject system size large
controlledexperience levelexperience levelexperience level beginnerbeginner
controlledexperience levelexperience levelexperience level advancedadvancedcontrolledbackgroundbackground academiaacademiaacademiacontrolledbackgroundbackground industryindustryindustry
Variables of the experiment
dependentcorrectnesscorrectnesscorrectnesscorrectnesscorrectness
dependentcompletion timecompletion timecompletion timecompletion timecompletion time
independenttool CodeCityCodeCityCodeCityCodeCity
independenttool Eclipse + ExcelEclipse + ExcelEclipse + ExcelEclipse + Excelindependentobject system sizeobject system sizeobject system sizeobject system size mediumindependentobject system sizeobject system sizeobject system sizeobject system size large
controlledexperience levelexperience levelexperience level beginnerbeginner
controlledexperience levelexperience levelexperience level advancedadvancedcontrolledbackgroundbackground academiaacademiaacademiacontrolledbackgroundbackground industryindustryindustry
FindBugs
1,320 classes
93,310 LOC
Azureus
4,656 classes
454,387 LOC
Variables of the experiment
dependentcorrectnesscorrectnesscorrectnesscorrectnesscorrectness
dependentcompletion timecompletion timecompletion timecompletion timecompletion time
independenttool CodeCityCodeCityCodeCityCodeCity
independenttool Eclipse + ExcelEclipse + ExcelEclipse + ExcelEclipse + Excelindependentobject system sizeobject system sizeobject system sizeobject system size mediumindependentobject system sizeobject system sizeobject system sizeobject system size large
controlledexperience levelexperience levelexperience level beginnerbeginner
controlledexperience levelexperience levelexperience level advancedadvancedcontrolledbackgroundbackground academiaacademiaacademiacontrolledbackgroundbackground industryindustryindustry
Variables of the experiment
dependentcorrectnesscorrectnesscorrectnesscorrectnesscorrectness
dependentcompletion timecompletion timecompletion timecompletion timecompletion time
independenttool CodeCityCodeCityCodeCityCodeCity
independenttool Eclipse + ExcelEclipse + ExcelEclipse + ExcelEclipse + Excelindependentobject system sizeobject system sizeobject system sizeobject system size mediumindependentobject system sizeobject system sizeobject system sizeobject system size large
controlledexperience levelexperience levelexperience level beginnerbeginner
controlledexperience levelexperience levelexperience level advancedadvancedcontrolledbackgroundbackground academiaacademiaacademiacontrolledbackgroundbackground industryindustryindustry
The experiment’s design
between-subjectsrandomized-block
The experiment’s design
Tool
CodeCityT1 large
SizeTool
CodeCity
T2 mediumSizeTool
Ecl+ExclT3 large
SizeTool
Ecl+Excl
T4 medium
Size
between-subjectsrandomized-block
The experiment’s designbackground academiaacademia industryindustry
experience beginner advanced beginner advanced
B1 B2 B3 B4
Tool
CodeCityT1 large
SizeTool
CodeCity
T2 mediumSizeTool
Ecl+ExclT3 large
SizeTool
Ecl+Excl
T4 medium
Size
between-subjectsrandomized-block
The experiment’s designbackground academiaacademia industryindustry
experience beginner advanced beginner advanced
B1 B2 B3 B4
Tool
CodeCityT1 large
SizeTool
CodeCity
T2 mediumSizeTool
Ecl+ExclT3 large
SizeTool
Ecl+Excl
T4 medium
Size
between-subjectsrandomized-block
Execution
Experimental runs
Experimental runs
timeday 1
training session(1 hour)
Experimental runs
e1
c1
timeday 1
training session(1 hour)
experiment session(2 hours)
Experimental runs
e1
c1
e2
c2
timeday 1 day 2
training session(1 hour)
experiment session(2 hours)
Experimental runs
e1
c1
e2
c2
e3
timeday 1 day 2 day 3
training session(1 hour)
experiment session(2 hours)
Experimental runs
e1
c1
e2
c2
e3
c4
timeday 1 day 2 day 3 day 4
training session(1 hour)
experiment session(2 hours)
Testing the waters20092009
November December
18 24 25 2 9
Lugano111
31
11
Timeline of the experiment20092009 2010201020102010
November December January February ... April
18 24 25 2 9 21 28 5 8 14 28 18 22 24 25 14
Bern
Bologna
Antwerp
Lugano111
31
11
21
1 1 13
11 1
61
56
46
1
Timeline of the experiment20092009 2010201020102010
November December January February ... April
18 24 25 2 9 21 28 5 8 14 28 18 22 24 25 14
Bern
Bologna
Antwerp
Lugano111
31
11
21
1 1 13
11 1
61
56
46
1
remotesessions
remotesession
Treatments and subjects
academiaacademia industry
beginner advanced advanced
CodeCity
large 2 2 6 10CodeCity
medium 3 2 7 12
Ecl+Excl
large 2 3 3 8Ecl+Excl
medium 2 5 4 11
9 12 20 41
Collecting raw data
Collecting raw data
solution
Collecting raw datacompletion time
Controlling time
Controlling time
common time
Controlling time
info on subjects
common time
Name (Task): Remaining time
Assessing correctness1 T2: Findbugs, analyzed with CodeCity
A1
Dispersed. [1pt]
A2.1
Localized [0.5pts]in package edu.umd.cs.findbugs.detect [0.5pts].
A2.2
Dispersedin the following (max. 5) packages [0.2pts for each]:
• edu.umd.cs.findbugs
• edu.umd.cs.findbugs.anttask
• edu.umd.cs.findbugs.ba
• edu.umd.cs.findbugs.ba.deref
• edu.umd.cs.findbugs.ba.jsr305
• edu.umd.cs.findbugs.ba.npe
• edu.umd.cs.findbugs.ba.vna
• edu.umd.cs.findbugs.bcel
• edu.umd.cs.findbugs.classfile
• edu.umd.cs.findbugs.classfile.analysis
• edu.umd.cs.findbugs.classfile.engine
• edu.umd.cs.findbugs.classfile.impl
• edu.umd.cs.findbugs.cloud
• edu.umd.cs.findbugs.cloud.db
• edu.umd.cs.findbugs.detect
• edu.umd.cs.findbugs.gui
• edu.umd.cs.findbugs.gui2
• edu.umd.cs.findbugs.jaif
• edu.umd.cs.findbugs.model
• edu.umd.cs.findbugs.visitclass
• edu.umd.cs.findbugs.workflow
A3: Impact Analysis
Multiple locations.There are 40/41 [0.5pts] classesdefined in the following 3 packages [1/6pts for each]:
• edu.umd.cs.findbugs
• edu.umd.cs.findbugs.bcel
• edu.umd.cs.findbugs.detect
A4.1
The 3 classes with the highest number of methods are [ 13pts each correctly placed and 16pts each misplaced]:
1. class AbstractFrameModelingVisitordefined in package edu.umd.cs.findbugs.bacontains 195 methods;
2. class MainFramedefined in package edu.umd.cs.findbugs.gui2contains 119 methods;
3. class BugInstancedefined in package edu.umd.cs.findbugscontains 118 methodsorclass TypeFrameModelingVisitordefined in package edu.umd.cs.findbugs.ba.typecontains 118 methods.
A4.2
The 3 classes with the highest average number of lines of code per method are [ 13pts each correctly placed and 16pts each
misplaced]:
1. class DefaultNullnessAnnotationsdefined in package edu.umd.cs.findbugs.bahas an average of 124 lines of code per method;
2. class DBCloud.PopulateBugsdefined in package edu.umd.cs.findbugs.cloud.dbhas an average of 114.5 lines of code per method;
3. class BytecodeScannerdefined in package edu.umd.cs.findbugs.bahas an average of 80.75 lines of code per method.
B1.1
The package with the highest percentage of god classes in the system isedu.umd.cs.findbugs.ba.deref [0.8pts]which contains 1 [0.1pts] god classesout of a total of 3 [0.1pts] classes.
B1.2
The god class containing the largest number of methods in the system isclass MainFrame [0.8pts]defined in package edu.umd.cs.findbugs.gui2 [0.1pts]which contains 119 [0.1pts] methods.
B2.1
The dominant class-level design problem isDataClass [0.5pts]which affects a number of 67 [0.5pts] classes.
oracles
Assessing correctness1 T2: Findbugs, analyzed with CodeCity
A1
Dispersed. [1pt]
A2.1
Localized [0.5pts]in package edu.umd.cs.findbugs.detect [0.5pts].
A2.2
Dispersedin the following (max. 5) packages [0.2pts for each]:
• edu.umd.cs.findbugs
• edu.umd.cs.findbugs.anttask
• edu.umd.cs.findbugs.ba
• edu.umd.cs.findbugs.ba.deref
• edu.umd.cs.findbugs.ba.jsr305
• edu.umd.cs.findbugs.ba.npe
• edu.umd.cs.findbugs.ba.vna
• edu.umd.cs.findbugs.bcel
• edu.umd.cs.findbugs.classfile
• edu.umd.cs.findbugs.classfile.analysis
• edu.umd.cs.findbugs.classfile.engine
• edu.umd.cs.findbugs.classfile.impl
• edu.umd.cs.findbugs.cloud
• edu.umd.cs.findbugs.cloud.db
• edu.umd.cs.findbugs.detect
• edu.umd.cs.findbugs.gui
• edu.umd.cs.findbugs.gui2
• edu.umd.cs.findbugs.jaif
• edu.umd.cs.findbugs.model
• edu.umd.cs.findbugs.visitclass
• edu.umd.cs.findbugs.workflow
A3: Impact Analysis
Multiple locations.There are 40/41 [0.5pts] classesdefined in the following 3 packages [1/6pts for each]:
• edu.umd.cs.findbugs
• edu.umd.cs.findbugs.bcel
• edu.umd.cs.findbugs.detect
A4.1
The 3 classes with the highest number of methods are [ 13pts each correctly placed and 16pts each misplaced]:
1. class AbstractFrameModelingVisitordefined in package edu.umd.cs.findbugs.bacontains 195 methods;
2. class MainFramedefined in package edu.umd.cs.findbugs.gui2contains 119 methods;
3. class BugInstancedefined in package edu.umd.cs.findbugscontains 118 methodsorclass TypeFrameModelingVisitordefined in package edu.umd.cs.findbugs.ba.typecontains 118 methods.
A4.2
The 3 classes with the highest average number of lines of code per method are [ 13pts each correctly placed and 16pts each
misplaced]:
1. class DefaultNullnessAnnotationsdefined in package edu.umd.cs.findbugs.bahas an average of 124 lines of code per method;
2. class DBCloud.PopulateBugsdefined in package edu.umd.cs.findbugs.cloud.dbhas an average of 114.5 lines of code per method;
3. class BytecodeScannerdefined in package edu.umd.cs.findbugs.bahas an average of 80.75 lines of code per method.
B1.1
The package with the highest percentage of god classes in the system isedu.umd.cs.findbugs.ba.deref [0.8pts]which contains 1 [0.1pts] god classesout of a total of 3 [0.1pts] classes.
B1.2
The god class containing the largest number of methods in the system isclass MainFrame [0.8pts]defined in package edu.umd.cs.findbugs.gui2 [0.1pts]which contains 119 [0.1pts] methods.
B2.1
The dominant class-level design problem isDataClass [0.5pts]which affects a number of 67 [0.5pts] classes.
oracles
blinding
Results
Statistical test
two-way analysis of variance (ANOVA)95% confidence interval
0
3
4
5
6
7
8
medium large
2
1
Correctness
Does the use of CodeCity increase the correctness of the solutions to program comprehension tasks, compared to non-visual exploration tools, regardless of the object system size?
Ecl+ExclCodeCity
0
3
4
5
6
7
8
medium large
2
1
Correctness
Does the use of CodeCity increase the correctness of the solutions to program comprehension tasks, compared to non-visual exploration tools, regardless of the object system size?
24.26%more correct with CodeCity
large effect size (d=0.89)
Ecl+ExclCodeCity
Correctness
24.26%more correct with CodeCity
Does the use of CodeCity increase the correctness of the solutions to program comprehension tasks, compared to non-visual exploration tools, regardless of the object system size?
large effect size (d=0.89)
2
3
4
5
6
7
8
Ecl+ExclCodeCity
medium large
Completion time
Does the use of CodeCity reduce the time needed to solve program comprehension tasks, compared to non-visual exploration tools, regardless of the object system size?
Ecl+ExclCodeCity
0
30
40
50
60
medium large
20
10
Completion time
Does the use of CodeCity reduce the time needed to solve program comprehension tasks, compared to non-visual exploration tools, regardless of the object system size?
12.01%less time with CodeCity
moderate effect size (d=0.63)
Ecl+ExclCodeCity
0
30
40
50
60
medium large
20
10
Completion time
12.01%less time with CodeCity
Does the use of CodeCity reduce the time needed to solve program comprehension tasks, compared to non-visual exploration tools, regardless of the object system size?
moderate effect size (d=0.63)
Ecl+ExclCodeCity
20
30
40
50
60
medium large
after the first round
CodeCity
vs
Ecl+Excl+24% correctness-12% completion time
Replicability193
A.5 Data
Code
Treatment
Blocking Criteria
Number ToolSystem size Background Experience
IA011 CodeCity
largeindustry
advanced
IA021 CodeCity
largeindustry
advanced
IA031 CodeCity
largeindustry
advanced
IA041 CodeCity
largeindustry
advanced
AB011 CodeCity
largeacademy
beginner
AB021 CodeCity
largeacademy
beginner
IA051 CodeCity
largeindustry
advanced
AA011 CodeCity
largeacademy
advanced
AA021 CodeCity
largeacademy
advanced
IA061 CodeCity
largeindustry
advanced
IA072 CodeCity
mediumindustry
advanced
IA082 CodeCity
mediumindustry
advanced
IA092 CodeCity
mediumindustry
advanced
AB032 CodeCity
mediumacademy
beginner
IA102 CodeCity
mediumindustry
advanced
IA112 CodeCity
mediumindustry
advanced
IA122 CodeCity
mediumindustry
advanced
AB042 CodeCity
mediumacademy
beginner
AA032 CodeCity
mediumacademy
advanced
AB052 CodeCity
mediumacademy
beginner
AA042 CodeCity
mediumacademy
advanced
IA132 CodeCity
mediumindustry
advanced
IA143 Ecl+Excl large
industryadvanced
AB063 Ecl+Excl large
academybeginner
AB073 Ecl+Excl large
academybeginner
AA053 Ecl+Excl large
academyadvanced
AA063 Ecl+Excl large
academyadvanced
AA073 Ecl+Excl large
academyadvanced
IA153 Ecl+Excl large
industryadvanced
IA163 Ecl+Excl large
industryadvanced
IA014 Ecl+Excl medium
industryadvanced
IA184 Ecl+Excl medium
industryadvanced
IA194 Ecl+Excl medium
industryadvanced
AB084 Ecl+Excl medium
academybeginner
AB094 Ecl+Excl medium
academybeginner
AA104 Ecl+Excl medium
academyadvanced
AA114 Ecl+Excl medium
academyadvanced
AA124 Ecl+Excl medium
academyadvanced
AA134 Ecl+Excl medium
academyadvanced
AA144 Ecl+Excl medium
academyadvanced
IA204 Ecl+Excl medium
industryadvanced
Table A.2. The assignment of the subjects to treatments and blocks
174
A.4 Task Solution OraclesA.4.1 T1: Azureus, analyzed with CodeCityA1
EitherThere are no unit tests in the system [1pt],orCentralized in a single package hierarchy whose root is in org.gudy.azureus2.ui.console.multiuser
[1pt]. Since there is only one test class (i.e., TestUserManager), if they don’t give the full correct
answer, the answer is completely wrong.A2.1
Dispersed [0pts otherwise]in the following (max. 5) packages [0.2pts for each]:• com.aelitis.azureus.core• com.aelitis.azureus.core.content• com.aelitis.azureus.core.download• com.aelitis.azureus.core.impl• com.aelitis.azureus.core.lws• com.aelitis.azureus.core.peermanager.peerdb• com.aelitis.azureus.core.stats
• com.aelitis.azureus.core.torrent• com.aelitis.azureus.plugins.net.buddy• com.aelitis.azureus.plugins.net.buddy.swt• com.aelitis.azureus.plugins.net.buddy.tracker• com.aelitis.azureus.plugins.removerules• com.aelitis.azureus.plugins.sharing.hoster• com.aelitis.azureus.plugins.startstoprules.defaultplugin• com.aelitis.azureus.plugins.tracker.dht
• com.aelitis.azureus.plugins.tracker.local• com.aelitis.azureus.plugins.tracker.peerauth• com.aelitis.azureus.ui.swt.content.columns• com.aelitis.azureus.ui.swt.shells.main• com.aelitis.azureus.util
174
A.4 Task Solution OraclesA.4.1 T1: Azureus, analyzed with CodeCityA1
EitherThere are no unit tests in the system [1pt],orCentralized in a single package hierarchy whose root is in org.gudy.azureus2.ui.console.multiuser
[1pt]. Since there is only one test class (i.e., TestUserManager), if they don’t give the full correctanswer, the answer is completely wrong.
A2.1
Dispersed [0pts otherwise]in the following (max. 5) packages [0.2pts for each]:• com.aelitis.azureus.core• com.aelitis.azureus.core.content• com.aelitis.azureus.core.download• com.aelitis.azureus.core.impl• com.aelitis.azureus.core.lws• com.aelitis.azureus.core.peermanager.peerdb• com.aelitis.azureus.core.stats• com.aelitis.azureus.core.torrent• com.aelitis.azureus.plugins.net.buddy• com.aelitis.azureus.plugins.net.buddy.swt• com.aelitis.azureus.plugins.net.buddy.tracker• com.aelitis.azureus.plugins.removerules• com.aelitis.azureus.plugins.sharing.hoster• com.aelitis.azureus.plugins.startstoprules.defaultplugin• com.aelitis.azureus.plugins.tracker.dht
• com.aelitis.azureus.plugins.tracker.local• com.aelitis.azureus.plugins.tracker.peerauth• com.aelitis.azureus.ui.swt.content.columns• com.aelitis.azureus.ui.swt.shells.main• com.aelitis.azureus.util
174 A.4 Task Solution Oracles
A.4.1 T1: Azureus, analyzed with CodeCityA1
EitherThere are no unit tests in the system [1pt],orCentralized in a single package hierarchy whose root is in org.gudy.azureus2.ui.console.multiuser[1pt]. Since there is only one test class (i.e., TestUserManager), if they don’t give the full correctanswer, the answer is completely wrong.
A2.1
Dispersed [0pts otherwise]in the following (max. 5) packages [0.2pts for each]:
• com.aelitis.azureus.core
• com.aelitis.azureus.core.content
• com.aelitis.azureus.core.download
• com.aelitis.azureus.core.impl
• com.aelitis.azureus.core.lws
• com.aelitis.azureus.core.peermanager.peerdb
• com.aelitis.azureus.core.stats
• com.aelitis.azureus.core.torrent
• com.aelitis.azureus.plugins.net.buddy
• com.aelitis.azureus.plugins.net.buddy.swt
• com.aelitis.azureus.plugins.net.buddy.tracker
• com.aelitis.azureus.plugins.removerules
• com.aelitis.azureus.plugins.sharing.hoster
• com.aelitis.azureus.plugins.startstoprules.defaultplugin
• com.aelitis.azureus.plugins.tracker.dht
• com.aelitis.azureus.plugins.tracker.local
• com.aelitis.azureus.plugins.tracker.peerauth
• com.aelitis.azureus.ui.swt.content.columns
• com.aelitis.azureus.ui.swt.shells.main
• com.aelitis.azureus.util
Introduction
The aim of this experiment is to compare tool efficiency in supporting software
practitioners analyzing medium to large-scale software systems.You will use CodeCity to analyze Azureus, a BitTorrent client written in Java.You are given maximum 100 minutes for solving 10 tasks (10 minutes per task). You are asked:• not to consult any other participant during the experiment;• to perform the tasks in the specified order;• to write down the current time each time before starting to read a task and once
after completing all the tasks;• to announce the experimenter that you are moving on to another task, in order
to reset your 10-minutes-per-task allocated timer;• not to return to earlier tasks because it affects the timing;• for each task, to fill in the required information. In the case of multiple choices
check the most appropriate answer and provide additional information, if
requested.
The experiment is concluded with a short debriefing questionnaire.
Thank you for participating in this experiment!! ! ! ! Richard Wettel, Michele Lanza, Romain Robbes
CodeCity Experiment
Participant:
T1
Introduction
The aim of this experiment is to compare tool efficiency in supporting software practitioners analyzing medium to large-scale software systems.You will use CodeCity to analyze Azureus, a BitTorrent client written in Java.You are given maximum 100 minutes for solving 10 tasks (10 minutes per task). You are asked:
• not to consult any other participant during the experiment;• to perform the tasks in the specified order;• to write down the current time each time before starting to read a task and once after completing all the tasks;• to announce the experimenter that you are moving on to another task, in order to reset your 10-minutes-per-task allocated timer;• not to return to earlier tasks because it affects the timing;• for each task, to fill in the required information. In the case of multiple choices check the most appropriate answer and provide additional information, if requested.
The experiment is concluded with a short debriefing questionnaire.
Thank you for participating in this experiment!! ! ! ! Richard Wettel, Michele Lanza, Romain Robbes
CodeCity Experiment
Participant:
T1
Introduction
The aim of this experiment is to compare tool efficiency in supporting software practitioners analyzing medium to large-scale software systems.
You will use CodeCity to analyze Azureus, a BitTorrent client written in Java.
You are given maximum 100 minutes for solving 10 tasks (10 minutes per task).
You are asked:• not to consult any other participant during the experiment;• to perform the tasks in the specified order;• to write down the current time each time before starting to read a task and once
after completing all the tasks;• to announce the experimenter that you are moving on to another task, in order
to reset your 10-minutes-per-task allocated timer;• not to return to earlier tasks because it affects the timing;• for each task, to fill in the required information. In the case of multiple choices
check the most appropriate answer and provide additional information, if requested.
The experiment is concluded with a short debriefing questionnaire.
Thank you for participating in this experiment!
! ! ! ! Richard Wettel, Michele Lanza, Romain Robbes
CodeCity Experiment
Participant:
T1
Introduction
The aim of this experiment is to compare tool efficiency in supporting software practitioners analyzing medium to large-scale software systems.
You will use CodeCity to analyze Azureus, a BitTorrent client written in Java.
You are given maximum 100 minutes for solving 10 tasks (10 minutes per task).
You are asked:• not to consult any other participant during the experiment;• to perform the tasks in the specified order;• to write down the current time each time before starting to read a task and once
after completing all the tasks;• to announce the experimenter that you are moving on to another task, in order
to reset your 10-minutes-per-task allocated timer;• not to return to earlier tasks because it affects the timing;• for each task, to fill in the required information. In the case of multiple choices
check the most appropriate answer and provide additional information, if requested.
The experiment is concluded with a short debriefing questionnaire.
Thank you for participating in this experiment!
! ! ! ! Richard Wettel, Michele Lanza, Romain Robbes
CodeCity Experiment
Participant:
T1
Pre-experimentquestionnaire Handouts
174 A.4 Task Solution Oracles
A.4.1 T1: Azureus, analyzed with CodeCity
A1
EitherThere are no unit tests in the system [1pt],orCentralized in a single package hierarchy whose root is in org.gudy.azureus2.ui.console.multiuser[1pt]. Since there is only one test class (i.e., TestUserManager), if they don’t give the full correctanswer, the answer is completely wrong.
A2.1
Dispersed [0pts otherwise]in the following (max. 5) packages [0.2pts for each]:
• com.aelitis.azureus.core
• com.aelitis.azureus.core.content
• com.aelitis.azureus.core.download
• com.aelitis.azureus.core.impl
• com.aelitis.azureus.core.lws
• com.aelitis.azureus.core.peermanager.peerdb
• com.aelitis.azureus.core.stats
• com.aelitis.azureus.core.torrent
• com.aelitis.azureus.plugins.net.buddy
• com.aelitis.azureus.plugins.net.buddy.swt
• com.aelitis.azureus.plugins.net.buddy.tracker
• com.aelitis.azureus.plugins.removerules
• com.aelitis.azureus.plugins.sharing.hoster
• com.aelitis.azureus.plugins.startstoprules.defaultplugin
• com.aelitis.azureus.plugins.tracker.dht
• com.aelitis.azureus.plugins.tracker.local
• com.aelitis.azureus.plugins.tracker.peerauth
• com.aelitis.azureus.ui.swt.content.columns
• com.aelitis.azureus.ui.swt.shells.main
• com.aelitis.azureus.util
Oracles Data
196
A.5Data
Code
Diffi
culty
Leve
l Per
Task
Tim
ePr
essu
re
A1
A2.1
A2.2
A3
A4.1
A4.2
B1.1
B1.2
B2.1
B2.2
IA01
diffi
cult
inte
rmed
iate
inte
rmed
iate
simpl
e
simpl
e
impo
ssib
le
diffi
cult
trivi
al
trivi
al
inte
rmed
iate
fair
amou
nt
IA02
fair
amou
nt
IA03
diffi
cult
inte
rmed
iate
inte
rmed
iate
diffi
cult
inte
rmed
iate
diffi
cult
simpl
e
simpl
e
simpl
e
impo
ssib
le
fair
amou
nt
IA04
simpl
e
simpl
e
simpl
e
simpl
e
inte
rmed
iate
impo
ssib
le
diffi
cult
simpl
e
trivi
al
diffi
cult
too
muc
h
AB01
trivi
al
trivi
al
trivi
al
simpl
e
simpl
e
inte
rmed
iate
diffi
cult
simpl
e
simpl
e
inte
rmed
iate
very
little
AB02
trivi
al
trivi
al
trivi
al
trivi
al
simpl
e
diffi
cult
impo
ssib
le
simpl
e
trivi
al
simpl
e
fair
amou
nt
IA05
simpl
e
simpl
e
simpl
e
simpl
e
inte
rmed
iate
impo
ssib
le
diffi
cult
diffi
cult
trivi
al
simpl
e
not s
om
uch
AA01
inte
rmed
iate
trivi
al
simpl
e
inte
rmed
iate
trivi
al
impo
ssib
le
impo
ssib
le
trivi
al
simpl
e
trivi
al
very
little
AA02
trivi
al
trivi
al
trivi
al
simpl
e
simpl
e
impo
ssib
le
diffi
cult
simpl
e
trivi
al
inte
rmed
iate
not s
om
uch
IA06
inte
rmed
iate
simpl
e
simpl
e
simpl
e
simpl
e
inte
rmed
iate
inte
rmed
iate
simpl
e
simpl
e
diffi
cult
fair
amou
nt
IA07
diffi
cult
inte
rmed
iate
inte
rmed
iate
simpl
e
simpl
e
diffi
cult
simpl
e
simpl
e
inte
rmed
iate
inte
rmed
iate
not s
om
uch
IA08
trivi
al
trivi
al
simpl
e
inte
rmed
iate
inte
rmed
iate
impo
ssib
le
diffi
cult
inte
rmed
iate
simpl
e
inte
rmed
iate
very
little
IA09
diffi
cult
simpl
e
simpl
e
inte
rmed
iate
inte
rmed
iate
diffi
cult
inte
rmed
iate
simpl
e
trivi
al
diffi
cult
not s
om
uch
AB03
simpl
e
simpl
e
simpl
e
simpl
e
simpl
e
inte
rmed
iate
inte
rmed
iate
simpl
e
simpl
e
inte
rmed
iate
very
little
IA10
simpl
e
simpl
e
simpl
e
simpl
e
diffi
cult
impo
ssib
le
inte
rmed
iate
inte
rmed
iate
inte
rmed
iate
diffi
cult
not s
om
uch
IA11
simpl
e
simpl
e
simpl
e
diffi
cult
inte
rmed
iate
diffi
cult
diffi
cult
simpl
e
simpl
e
diffi
cult
fair
amou
nt
IA12
simpl
e
simpl
e
simpl
e
inte
rmed
iate
inte
rmed
iate
diffi
cult
trivi
al
simpl
e
simpl
e
diffi
cult
none
AB04
trivi
al
trivi
al
trivi
al
simpl
e
inte
rmed
iate
diffi
cult
trivi
al
trivi
al
simpl
e
diffi
cult
very
little
AA03
trivi
al
trivi
al
trivi
al
diffi
cult
trivi
al
impo
ssib
le
diffi
cult
trivi
al
trivi
al
simpl
e
not s
om
uch
AB05
simpl
e
simpl
e
simpl
e
inte
rmed
iate
inte
rmed
iate
impo
ssib
le
diffi
cult
inte
rmed
iate
trivi
al
simpl
e
not s
om
uch
AA04
inte
rmed
iate
inte
rmed
iate
diffi
cult
diffi
cult
inte
rmed
iate
impo
ssib
le
inte
rmed
iate
inte
rmed
iate
diffi
cult
impo
ssib
le
fair
amou
nt
IA13
trivi
al
simpl
e
simpl
e
simpl
e
inte
rmed
iate
diffi
cult
inte
rmed
iate
inte
rmed
iate
inte
rmed
iate
diffi
cult
none
IA14
inte
rmed
iate
inte
rmed
iate
inte
rmed
iate
inte
rmed
iate
inte
rmed
iate
inte
rmed
iate
inte
rmed
iate
inte
rmed
iate
inte
rmed
iate
inte
rmed
iate
fair
amou
nt
AB06
simpl
e
simpl
e
simpl
e
inte
rmed
iate
inte
rmed
iate
diffi
cult
impo
ssib
le
inte
rmed
iate
inte
rmed
iate
inte
rmed
iate
fair
amou
nt
AB07
inte
rmed
iate
simpl
e
simpl
e
impo
ssib
le
trivi
al
diffi
cult
simpl
e
inte
rmed
iate
inte
rmed
iate
diffi
cult
too
muc
h
AA05
simpl
e
simpl
e
simpl
e
inte
rmed
iate
inte
rmed
iate
inte
rmed
iate
diffi
cult
inte
rmed
iate
inte
rmed
iate
inte
rmed
iate
fair
amou
nt
AA06
inte
rmed
iate
simpl
e
simpl
e
diffi
cult
trivi
al
inte
rmed
iate
simpl
e
simpl
e
inte
rmed
iate
diffi
cult
very
little
AA07
trivi
al
inte
rmed
iate
inte
rmed
iate
trivi
al
trivi
al
simpl
e
inte
rmed
iate
trivi
al
simpl
e
inte
rmed
iate
none
IA15
diffi
cult
inte
rmed
iate
inte
rmed
iate
diffi
cult
simpl
e
simpl
e
inte
rmed
iate
inte
rmed
iate
impo
ssib
le
impo
ssib
le
fair
amou
nt
IA16
inte
rmed
iate
inte
rmed
iate
inte
rmed
iate
impo
ssib
le
simpl
e
simpl
e
inte
rmed
iate
simpl
e
inte
rmed
iate
diffi
cult
fair
amou
nt
IA01
inte
rmed
iate
inte
rmed
iate
inte
rmed
iate
diffi
cult
trivi
al
simpl
e
impo
ssib
le
trivi
al
simpl
e
diffi
cult
not s
om
uch
IA18
inte
rmed
iate
simpl
e
inte
rmed
iate
diffi
cult
simpl
e
simpl
e
diffi
cult
inte
rmed
iate
simpl
e
diffi
cult
fair
amou
nt
IA19
simpl
e
simpl
e
inte
rmed
iate
simpl
e
trivi
al
simpl
e
simpl
e
simpl
e
inte
rmed
iate
impo
ssib
le
fair
amou
nt
AB08
inte
rmed
iate
inte
rmed
iate
simpl
e
inte
rmed
iate
trivi
al
trivi
al
diffi
cult
trivi
al
simpl
e
inte
rmed
iate
not s
om
uch
AB09
trivi
al
simpl
e
simpl
e
impo
ssib
le
simpl
e
simpl
e
inte
rmed
iate
inte
rmed
iate
simpl
e
diffi
cult
fair
amou
nt
AA10
simpl
e
inte
rmed
iate
inte
rmed
iate
inte
rmed
iate
trivi
al
trivi
al
simpl
e
simpl
e
inte
rmed
iate
impo
ssib
le
very
little
AA11
diffi
cult
simpl
e
inte
rmed
iate
inte
rmed
iate
simpl
e
simpl
e
simpl
e
inte
rmed
iate
inte
rmed
iate
impo
ssib
le
not s
om
uch
AA12
simpl
e
simpl
e
simpl
e
simpl
e
simpl
e
inte
rmed
iate
simpl
e
simpl
e
inte
rmed
iate
diffi
cult
not s
om
uch
AA13
simpl
e
inte
rmed
iate
inte
rmed
iate
diffi
cult
simpl
e
simpl
e
simpl
e
simpl
e
simpl
e
inte
rmed
iate
not s
om
uch
AA14
trivi
al
trivi
al
trivi
al
impo
ssib
le
trivi
al
trivi
al
diffi
cult
trivi
al
diffi
cult
impo
ssib
le
not s
om
uch
IA20
inte
rmed
iate
simpl
e
simpl
e
inte
rmed
iate
trivi
al
trivi
al
trivi
al
trivi
al
trivi
al
diffi
cult
not s
om
uch
Tabl
eA.
5.Th
esu
bjec
ts’pe
rcei
ved
time
pres
sure
and
task
diffi
culty
192
A.5Data
Code
Age
Job
Posit
ion
Expe
rienc
e Leve
l
Numbe
r ofYe
ars
OOP
Java
Eclip
se
Rev.E
ng.
OOP
Java
Eclip
seRe
v.Eng
.
IA01
30
Develo
per
know
ledge
able
adva
nced
know
ledge
able
begin
ner
7–10
7–10
4–6
1–3
IA02
34
Develo
per
adva
nced
adva
nced
know
ledge
able
know
ledge
able
7–10
4–6
1–3
4–6
IA03
42
CTO, D
evelo
per
expe
rt
know
ledge
able
begin
ner
know
ledge
able
>10
1–3
1–3
>10
IA04
37
Develo
per
adva
nced
adva
nced
know
ledge
able
begin
ner
7–10
7–10
4–6
1–3
AB01
21
Mas
terSt
uden
t
adva
nced
adva
nced
adva
nced
begin
ner
4–6
4–6
4–6
<1
AB02
21
Mas
terSt
uden
t
adva
nced
adva
nced
adva
nced
begin
ner
1–3
1–3
1–3
<1
IA05
29
Consu
ltant
, Ph.D
. Stu
dent
expe
rt
begin
ner
begin
ner
know
ledge
able
7–10
7–10
4–6
4–6
AA01
26
Ph.D
. Stu
dent
expe
rt
expe
rt
know
ledge
able
begin
ner
>10
>10
1–3
<1
AA02
26
Ph.D
. Stu
dent
expe
rt
adva
nced
know
ledge
able
know
ledge
able
4–6
1–3
1–3
1–3
IA06
35
Head of
IT
expe
rt
expe
rt
adva
nced
adva
nced
>10
>10
4–6
7–10
IA07
27
Softw
are En
ginee
r
know
ledge
able
know
ledge
able
begin
ner
know
ledge
able
7–10
7–10
1–3
4–6
IA08
25
Softw
are En
ginee
r
know
ledge
able
adva
nced
know
ledge
able
begin
ner
4–6
4–6
1–3
<1
IA09
32
Develo
pmen
t Lead
er,Re
sear
cher
adva
nced
begin
ner
none
adva
nced
7–10
7–10
<1
4–6
AB03
28
Stud
ent
know
ledge
able
know
ledge
able
begin
ner
begin
ner
4–6
1–3
1–3
1–3
IA10
39
Proje
ctM
anag
er
expe
rt
adva
nced
know
ledge
able
know
ledge
able
>10
7–10
7–10
4–6
IA11
38
Consu
ltant
, Sys
temM
anag
er/A
nalys
t
know
ledge
able
begin
ner
begin
ner
adva
nced
7–10
7–10
1–3
7–10
IA12
34
Senio
r Java
Archite
ct
expe
rt
expe
rt
adva
nced
adva
nced
>10
>10
>10
>10
AB04
22
Mas
terSt
uden
t
adva
nced
adva
nced
adva
nced
know
ledge
able
4–6
1–3
1–3
<1
AA03
22
Mas
terSt
uden
t
adva
nced
adva
nced
adva
nced
begin
ner
7–10
4–6
4–6
<1
AB05
22
Mas
terSt
uden
t
adva
nced
adva
nced
know
ledge
able
begin
ner
4–6
1–3
1–3
1–3
AA04
29
Ph.D
. Stu
dent
adva
nced
adva
nced
know
ledge
able
begin
ner
4–6
4–6
1–3
<1
IA13
32
Consu
ltant
expe
rt
know
ledge
able
know
ledge
able
expe
rt
7–10
4–6
1–3
7–10
IA14
31
Softw
are Arch
itect
adva
nced
know
ledge
able
know
ledge
able
begin
ner
7–10
7–10
1–3
1–3
AB06
23
Mas
terSt
uden
t
adva
nced
adva
nced
adva
nced
begin
ner
4–6
1–3
1–3
<1
AB07
23
Mas
terSt
uden
t
adva
nced
adva
nced
adva
nced
begin
ner
4–6
1–3
1–3
<1
AA05
30
Ph.D
. Stu
dent
adva
nced
adva
nced
adva
nced
know
ledge
able
7–10
7–10
7–10
4–6
AA06
26
Ph.D
. Stu
dent
expe
rt
know
ledge
able
adva
nced
expe
rt
7–10
7–10
4–6
4–6
AA07
30
Ph.D
. Stu
dent
adva
nced
adva
nced
know
ledge
able
know
ledge
able
7–10
7–10
1–3
1–3
IA15
40
Proje
ctM
anag
er
expe
rt
expe
rt
adva
nced
adva
nced
>10
>10
7–10
4–6
IA16
39
Softw
are Arch
itect
adva
nced
adva
nced
know
ledge
able
know
ledge
able
4–6
4–6
4–6
1–3
IA01
30
Develo
per
know
ledge
able
adva
nced
know
ledge
able
begin
ner
7–10
7–10
4–6
1–3
IA17
27
Softw
are En
ginee
r
know
ledge
able
adva
nced
know
ledge
able
begin
ner
4–6
4–6
4–6
<1
IA19
39
Consu
ltant
, Pro
ject M
anag
er,Arch
itect
expe
rt
expe
rt
know
ledge
able
adva
nced
>10
7–10
7–10
4–6
AB08
21
Mas
terSt
uden
t
adva
nced
adva
nced
adva
nced
begin
ner
1–3
1–3
1–3
<1
AB09
23
Ph.D
. Stu
dent
adva
nced
adva
nced
adva
nced
know
ledge
able
4–6
1–3
1–3
1–3
AA10
24
Ph.D
. Stu
dent
adva
nced
adva
nced
adva
nced
adva
nced
4–6
4–6
4–6
1–3
AA11
23
Ph.D
. Stu
dent
adva
nced
adva
nced
adva
nced
know
ledge
able
4–6
4–6
4–6
1–3
AA12
52
Profe
ssor
expe
rt
adva
nced
know
ledge
able
adva
nced
>10
>10
4–6
>10
AA13
28
Ph.D
. Stu
dent
adva
nced
adva
nced
know
ledge
able
know
ledge
able
4–6
4–6
4–6
1–3
AA14
24
Mas
terSt
uden
t
expe
rt
expe
rt
expe
rt
know
ledge
able
4–6
4–6
4–6
1–3
IA20
36
Develo
per
adva
nced
expe
rt
adva
nced
begin
ner
>10
7–10
7–10
1–3
Table
A.1.Th
e subje
cts’ p
erson
alinf
ormati
on, c
luster
edby
treatm
ent c
ombin
ation
s
195
A.5 Data
Code
Completion Time Per Task TotalCompl. Time
A1 A2.1 A2.2 A3 A4.1 A4.2 B1.1 B1.2 B2.1 (excl. A4.2)
IA01 8.80 4.32 5.38 10.00 7.28 4.42 9.85 1.90 1.02 52.97 48.55
IA02 2.42 6.08 1.75 8.58 5.33 10.00 5.25 2.92 2.33 44.67 34.67
IA03 6.25 6.92 2.42 9.43 5.62 9.37 2.55 2.95 1.83 47.33 37.97
IA04 9.08 6.00 4.00 6.08 7.92 10.00 10.00 1.33 1.08 55.50 45.50
AB01 1.33 4.83 3.92 4.75 6.25 10.00 6.92 2.25 2.58 42.83 32.83
AB02 10.00 5.58 1.67 4.25 8.08 10.00 9.42 2.58 1.67 53.25 43.25
IA05 6.33 3.67 1.33 4.75 3.50 10.00 3.75 1.58 2.17 37.08 27.08
AA01 9.67 4.42 2.08 8.75 3.17 10.00 5.17 1.33 1.42 46.00 36.00
AA02 8.33 7.17 2.50 10.00 5.00 9.75 8.33 3.25 1.33 55.67 45.92
IA06 10.00 5.92 4.42 10.00 4.17 10.00 7.33 1.33 1.50 54.67 44.67
IA07 7.25 5.67 6.25 6.33 4.58 10.00 2.42 1.42 2.33 46.25 36.25
IA08 2.67 2.25 2.95 4.55 3.75 10.00 3.33 4.00 1.17 34.67 24.67
IA09 10.00 3.67 3.00 10.00 5.25 7.83 3.67 2.50 1.42 47.33 39.50
AB03 6.33 2.00 9.33 5.00 5.00 8.50 7.17 1.50 1.25 46.08 37.58
IA10 3.25 3.67 7.33 6.83 3.83 7.75 3.08 1.75 8.08 45.58 37.83
IA11 3.83 2.40 5.92 7.17 4.00 6.92 7.42 2.83 1.75 42.23 35.32
IA12 3.75 2.67 4.17 5.58 5.17 10.00 5.75 2.00 1.25 40.33 30.33
AB04 2.67 3.92 2.58 3.67 5.58 9.75 2.00 0.50 3.83 34.50 24.75
AA03 2.50 3.17 3.75 10.00 3.17 6.33 4.08 1.25 2.58 36.83 30.50
AB05 5.50 4.67 4.33 5.58 5.83 10.00 9.58 1.83 1.67 49.00 39.00
AA04 7.08 3.67 5.17 6.33 3.50 7.83 6.00 1.25 2.83 43.67 35.83
IA13 3.00 4.67 2.33 4.75 3.25 10.00 4.83 1.67 2.08 36.58 26.58
IA14 6.67 9.00 2.42 10.00 4.50 5.83 5.33 1.83 4.17 49.75 43.92
AB06 5.67 5.75 2.05 6.95 10.00 10.00 10.00 3.85 2.00 56.27 46.27
AB07 8.75 9.33 4.67 10.00 6.83 10.00 10.00 3.67 2.67 65.92 55.92
AA05 7.00 6.08 3.75 8.42 5.25 4.42 10.00 3.17 6.67 54.75 50.33
AA06 6.83 2.00 3.67 6.58 3.25 6.75 5.33 1.58 1.58 37.58 30.83
AA07 3.67 3.67 2.50 2.17 2.92 5.17 2.75 3.33 1.83 28.00 22.83
IA15 9.75 8.83 5.08 10.00 4.33 10.00 5.08 1.75 7.83 62.67 52.67
IA16 10.00 7.42 4.33 9.50 4.00 6.50 8.50 2.00 4.50 56.75 50.25
IA01 2.55 3.90 4.38 9.20 4.03 4.30 9.92 2.10 2.93 43.32 39.02
IA18 5.28 5.13 4.58 9.82 4.72 3.98 9.77 4.13 3.12 50.53 46.55
IA19 3.33 3.83 4.50 3.50 3.83 5.08 6.58 1.92 5.58 38.17 33.08
AB08 6.08 1.08 10.00 8.83 3.50 5.92 9.42 3.58 1.92 50.33 44.42
AB09 5.83 4.83 4.33 10.00 3.42 6.00 2.00 1.92 1.50 39.83 33.83
AA10 3.17 6.08 5.08 7.33 8.00 3.33 4.83 1.17 2.33 41.33 38.00
AA11 6.17 4.08 6.08 3.83 3.50 5.17 3.67 1.67 2.92 37.08 31.92
AA12 6.75 4.75 3.92 4.75 5.25 3.33 4.42 1.42 3.00 37.58 34.25
AA13 7.00 7.00 10.00 10.00 5.92 5.67 6.33 1.42 5.42 58.75 53.08
AA14 6.33 1.50 3.33 10.00 2.83 3.25 9.17 2.42 4.50 43.33 40.08
IA20 6.83 5.58 4.75 8.00 2.33 3.42 2.42 1.42 1.33 36.08 32.67
Table A.4. The subjects’ task completion time, in minutes
194 A.5 Data
CodeCorrectness Per Task
TotalCorrectness
A1 A2.1 A2.2 A3 A4.1 A4.2 B1.1 B1.2 B2.1 (excl. A4.2)IA01 0.00 1.00 1.00 1.00 1.00 0.00 0.00 1.00 1.00 6.00 6.00IA02 1.00 0.80 0.50 1.00 1.00 0.00 0.00 1.00 1.00 6.30 6.30IA03 0.00 0.00 1.00 0.00 1.00 0.00 0.00 1.00 0.00 3.00 3.00IA04 0.00 1.00 0.00 1.00 1.00 0.00 0.00 1.00 1.00 5.00 5.00AB01 0.00 1.00 1.00 0.80 1.00 0.00 0.00 1.00 1.00 5.80 5.80AB02 0.00 1.00 1.00 0.70 1.00 0.00 0.00 0.00 1.00 4.70 4.70IA05 0.00 1.00 1.00 0.20 1.00 0.00 0.00 1.00 1.00 5.20 5.20AA01 0.00 0.80 1.00 0.40 1.00 0.00 0.00 1.00 0.00 4.20 4.20AA02 0.00 1.00 1.00 0.00 1.00 0.00 1.00 1.00 1.00 6.00 6.00IA06 0.00 1.00 0.00 0.30 1.00 0.00 0.00 1.00 1.00 4.30 4.30IA07 1.00 1.00 1.00 0.17 1.00 0.00 0.00 1.00 1.00 6.17 6.17IA08 1.00 0.50 1.00 0.83 1.00 0.00 1.00 1.00 1.00 7.33 7.33IA09 1.00 1.00 1.00 1.00 1.00 0.00 1.00 1.00 1.00 8.00 8.00AB03 1.00 1.00 1.00 1.00 1.00 0.00 1.00 1.00 1.00 8.00 8.00IA10 1.00 1.00 1.00 0.17 1.00 0.00 0.00 1.00 1.00 6.17 6.17IA11 1.00 1.00 1.00 0.00 1.00 0.16 0.00 0.00 1.00 5.16 5.00IA12 1.00 1.00 1.00 1.00 1.00 0.00 1.00 1.00 1.00 8.00 8.00AB04 1.00 1.00 1.00 0.67 1.00 0.00 0.00 1.00 1.00 6.67 6.67AA03 1.00 1.00 0.80 0.33 1.00 0.16 0.00 1.00 1.00 6.29 6.13AB05 1.00 1.00 1.00 1.00 1.00 0.00 0.00 1.00 1.00 7.00 7.00AA04 1.00 1.00 1.00 0.50 1.00 0.00 0.00 1.00 1.00 6.50 6.50IA13 1.00 1.00 1.00 0.83 1.00 1.00 0.00 0.00 1.00 6.83 5.83IA14 0.00 0.80 0.33 0.30 1.00 0.67 0.00 1.00 1.00 5.10 4.43AB06 1.00 1.00 1.00 0.00 0.00 0.00 0.00 1.00 0.00 4.00 4.00AB07 0.00 0.60 0.67 0.00 0.00 0.00 0.00 0.00 1.00 2.27 2.27AA05 1.00 1.00 1.00 0.00 1.00 1.00 0.00 1.00 1.00 7.00 6.00AA06 0.00 1.00 0.00 0.00 1.00 1.00 0.00 0.00 1.00 4.00 3.00AA07 1.00 1.00 0.00 0.00 1.00 1.00 0.00 1.00 0.00 5.00 4.00IA15 1.00 0.80 0.00 0.00 1.00 1.00 0.00 1.00 0.00 4.80 3.80IA16 0.00 0.00 0.67 0.00 1.00 0.67 0.00 1.00 1.00 4.34 3.67IA01 1.00 1.00 1.00 0.50 1.00 1.00 0.00 1.00 1.00 7.50 6.50IA18 1.00 1.00 1.00 0.50 1.00 1.00 0.00 1.00 1.00 7.50 6.50IA19 0.00 1.00 0.60 0.00 1.00 1.00 0.00 1.00 0.00 4.60 3.60AB08 0.00 1.00 0.40 0.38 1.00 1.00 1.00 1.00 1.00 6.78 5.78AB09 1.00 1.00 0.60 0.50 1.00 1.00 0.00 1.00 1.00 7.10 6.10AA10 1.00 1.00 0.80 0.38 1.00 1.00 0.00 1.00 1.00 7.18 6.18AA11 1.00 1.00 0.80 0.00 1.00 1.00 0.00 1.00 1.00 6.80 5.80AA12 0.00 0.50 0.00 0.00 1.00 1.00 0.00 1.00 1.00 4.50 3.50AA13 0.00 1.00 0.00 0.25 1.00 1.00 0.00 1.00 1.00 5.25 4.25AA14 1.00 1.00 0.60 0.00 1.00 1.00 0.90 1.00 1.00 7.50 6.50IA20 0.00 1.00 1.00 0.38 1.00 1.00 0.00 1.00 1.00 6.38 5.38
Table A.3. The correctness of the subjects’ solutions to the tasks
technical report 2010
Conclusions
Summary
Summarylist ofdesigndesiderata
Summarylist ofdesigndesiderata
6 months4 locations41 subjects
Summarylist ofdesigndesiderata
6 months4 locations41 subjects
-12.01%
+24.26%correctness
completiontime
Take-away messageVisualizing software systems as
cities is a viable alternative to the state of the practice approaches for software exploration.
Radu Marinescu, Alberto Bacchelli, Mircea Lungu, Lile Hattori, Andrea Reber, Kristina Gulordava, Francesco Rigotti, Vitezslav Humpa, Luca Della Toffola, Marcin Nowak, Aida Hota, Alessio Böckmann, Mattia Filippi, Fabio Parentela, Enrico Baraldi, Giacomo Benvenuti, Andrea Bolognesi, Marco Vit, Marco Brizi, Alberto Brandolini, Renzo Borgatti, Michele Finelli, Yuri Valentini, Fausto Fabbioni, Lorenzo Bragaglia, Riccardo Contri, Fabrizio Pinto, Paolo Gruppioni, Pasquale Granato, Andrea Chiodoni, Mario Fusco, Andrea Polci, Gianfranco Tognana, Glenn Van Loon, Tom Pauwaert, Yorik Gerlo, Serge Demeyer, Quinten Soetens, Bart Smets, Dieter De Hen, Ahmed Lamkanfi, Wim Van Eynde, Nick Baetens, Matthias De Cock, Tim Molderez, Oscar Nierstrasz, Tudor Gîrba, Adrian Lienhard, Fabrizio Perin, Erwann Wernli, Lukas Renggli, Niko Schwarz, David Röthlisberger, Jorge Ressia, Camillo Bruni, Toon Verwaest, Adrian Kuhn, and
We thank
Radu Marinescu, Alberto Bacchelli, Mircea Lungu, Lile Hattori, Andrea Reber, Kristina Gulordava, Francesco Rigotti, Vitezslav Humpa, Luca Della Toffola, Marcin Nowak, Aida Hota, Alessio Böckmann, Mattia Filippi, Fabio Parentela, Enrico Baraldi, Giacomo Benvenuti, Andrea Bolognesi, Marco Vit, Marco Brizi, Alberto Brandolini, Renzo Borgatti, Michele Finelli, Yuri Valentini, Fausto Fabbioni, Lorenzo Bragaglia, Riccardo Contri, Fabrizio Pinto, Paolo Gruppioni, Pasquale Granato, Andrea Chiodoni, Mario Fusco, Andrea Polci, Gianfranco Tognana, Glenn Van Loon, Tom Pauwaert, Yorik Gerlo, Serge Demeyer, Quinten Soetens, Bart Smets, Dieter De Hen, Ahmed Lamkanfi, Wim Van Eynde, Nick Baetens, Matthias De Cock, Tim Molderez, Oscar Nierstrasz, Tudor Gîrba, Adrian Lienhard, Fabrizio Perin, Erwann Wernli, Lukas Renggli, Niko Schwarz, David Röthlisberger, Jorge Ressia, Camillo Bruni, Toon Verwaest, Adrian Kuhn, and
We thank
None of the participants has been harmed during the experiment.
http://creativecommons.org/licenses/by/3.0/
Richard Wettelhttp://www.inf.usi.ch/phd/wettel