Embed Size (px)
Citation preview
Chianti: A Tool for Change Impact Analysis of JavaPrograms
Xiaoxia Ren1, Fenil Shah2, Frank Tip3, Barbara G. Ryder1, and Ophelia Chesley1∗
Division of Computer and Information Sciences1 IBM Software Group2 IBM T.J. Watson Research Center3
Rutgers University 17 Skyline Drive P.O. Box 704110 Frelinghuysen Road Hawthorne, NY 10532, USA Yorktown Heights, NY 10598, USA
Piscataway, NJ 08854-8019, USA [email protected] [email protected]{xren,ryder,ochesley}@cs.rutgers.edu
ABSTRACTThis paper reports on the design and implementation ofChianti, a change impact analysis tool for Java that is imple-mented in the context of the Eclipse environment. Chiantianalyzes two versions of an application and decomposes theirdifference into a set of atomic changes. Change impact isthen reported in terms of affected (regression or unit) testswhose execution behavior may have been modified by theapplied changes. For each affected test, Chianti also deter-mines a set of affecting changes that were responsible forthe test’s modified behavior. This latter step of isolatingthe changes that induce the failure of one specific test fromthose changes that only affect other tests can be used as adebugging technique in situations where a test fails unex-pectedly after a long editing session.
We evaluated Chianti on a year (2002) of CVS data fromM. Ernst’s Daikon system, and found that, on average, 52%of Daikon’s unit tests are affected. Furthermore, each af-fected unit test, on average, is affected by only 3.95% ofthe atomic changes. These findings suggest that our changeimpact analysis is a promising technique for assisting devel-opers with program understanding and debugging.
Categories and Subject DescriptorsD.2.5 [Software Engineering]: Testing and Debugging;D.2.6 [Software Engineering]: Programming Environ-ments; F.3.2 [Logics and Meanings of Programs]: Se-mantics of Programming Languages—Program analysis
General TermsAlgorithms, Measurement, Languages, Reliability
∗This research was supported by NSF grant CCR-0204410and in part by REU supplement CCR-0331797.
Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.OOPSLA’04,Oct. 24-28, 2004, Vancouver, British Columbia, Canada.Copyright 2004 ACM 1-58113-831-8/04/0010 ...$5.00.
KeywordsChange impact analysis, regression test, unit test, analysisof object-oriented programs
1. INTRODUCTIONThe extensive use of subtyping and dynamic dispatch in
object-oriented programming languages make it difficult tounderstand value flow through a program. For example,adding the creation of an object may affect the behavior ofvirtual method calls that are not lexically near the alloca-tion site. Also, adding a new method definition that over-rides an existing method can have a similar non-local effect.This nonlocality of change impact is qualitatively differentand more important for object-oriented programs than forimperative ones (e.g., in C programs a precise call graph canbe derived from syntactic information alone, except for thetypically few calls through function pointers [14]).
Change impact analysis [3, 13, 15, 21, 16] consists of acollection of techniques for determining the effects of sourcecode modifications, and can improve programmer produc-tivity by: (i) allowing programmers to experiment with dif-ferent edits, observe the code fragments that they affect,and use this information to determine which edit to selectand/or how to augment test suites, (ii) reducing the amountof time and effort needed in running regression1 tests, by de-termining that some tests are guaranteed not to be affectedby a given set of changes, and (iii) reducing the amount oftime and effort spent in debugging, by determining a safeapproximation of the changes responsible for a given test’sfailure [21, 19].
The change impact analysis method presented in this pa-per presumes the existence of a suite T of regression testsassociated with a Java program and access to the originaland edited versions of the code. Our analysis comprises thefollowing steps:
1. A source code edit is analyzed to obtain a set of in-terdependent atomic changes A, whose granularity is(roughly) at the method level. These atomic changesinclude all possible effects of the edit on dynamic dis-patch.
2. Then, a call graph is constructed for each test in T .Our method can use either dynamic call graphs thathave been obtained by tracing the execution of the
1In the rest of this paper, we will use the term “regressiontest” to refer to unit tests and other regression tests.
tests, or static call graphs that have been constructedby a static analysis engine. In this paper, we use dy-namic call graphs2.
3. For a given set T of regression tests, the analysis de-termines a subset T ′ of T that is potentially affectedby the changes in A, by correlating the changes in Aagainst the call graphs for the tests in T in the originalversion of the program.
4. Finally, for a given test ti ∈ T ′, the analysis can de-termine a subset A′ of A that contains all the changesthat may have affected the behavior of ti. This is ac-complished by constructing a call graph for ti in theedited version of the program, and correlating that callgraph with the changes in A.
The primary goal of our research is to provide programmerswith tool support that can help them understand why a testis suddenly failing after a long editing session by isolatingthe changes responsible for the failure.
There are some interesting similarities and differences be-tween the work presented in this paper, and previous workon regression test selection and change impact analysis.Step 3 above is similar in spirit to previous work on regres-sion test selection. However, unlike previous approaches ourtechnique does not rely on a pairwise comparison of high-level program representations such as control flow graphs(see, e.g. [20]) or Java InterClass Graphs [10]. Our workdiffers from previous approaches for dynamic change im-pact analysis [13, 15, 16] in the sense that these previousapproaches are primarily concerned with the problem of de-termining a subset of the methods in a program that wereaffected by a given set of changes. In contrast, step 4 ofour technique is concerned with the problem of isolating asubset of the changes that affect a given test. In addition,our approach decomposes the code edit into a set of seman-tically meaningful, interdependent ’atomic changes’ whichcan be used to generate intermediate program versions, inorder to investigate the cause of unexpected test behavior.These, and other connections to related work, will be furtherexplored in Section 6.
This paper reports on the engineering of Chianti, a proto-type change impact analysis tool, and its validation againstthe 2002 revision history (taken from the developers’ CVSrepository) of Daikon, a realistic Java system developed byM. Ernst et al. [7]. Essentially, in this initial study wesubstituted CVS updates obtained at intervals throughoutthe year for programmer edits, thus acquiring enough datato make some initial conclusions about our approach. Wepresent both data measuring the overall effectiveness of theanalysis and some case studies of individual CVS updates.Since the primary goal of our research has been to assist pro-grammers during development, Chianti has been integratedclosely with Eclipse, a widely used open-source developmentenvironment for Java (see www.eclipse.org).
The main contributions of this research are as follows:
• Demonstration of the utility of the basic change impactanalysis framework of [21], by implementing a proof-of-concept prototype, Chianti, and applying it to Daikon,a moderate-sized Java system built by others.
2Our previous study used static call graphs [19].
• Extension of the originally specified techniques [21] tohandle the entire Java language, including such con-structs as anonymous classes and overloading. Thiswork entailed extension of the model of atomic changesand their interdependences.
• Experimental validation of the utility of change im-pact analysis by determining the percentages of af-fected tests and affecting changes for 40 versions ofDaikon in 2002. For the 39 sets of changes betweenthese versions, we found that, on average, 52% of thetests are potentially affected. Moreover, for each po-tentially affected test, on average, only 3.95% of theatomic changes affected it. This is a promising resultwith regard to the utility of our technique for enhanc-ing program understanding and debugging.
In Section 2, we give intuition about our approach throughan example. In Section 3, the model of atomic changes isdiscussed, as well as engineering issues arising from handlingJava constructs that were previously not modeled. The im-plementation of Chianti is described in Section 4. Section 5describes the experimental setup and presents the empiricalfindings of the Daikon study. Related work and conclusionsare summarized in Sections 6 and 7, respectively.
2. OVERVIEW OF APPROACHThis section gives an informal overview of the change im-
pact analysis methodology originally presented in [21]. Ourapproach first determines, given two versions of a programand a set of tests that execute parts of the program, the af-fected tests whose behavior may have changed. Our methodis safe [20] in the sense that this set of affected tests containsat least every test whose behavior may have been affected.
Then, in a second step, for each test whose behavior wasaffected, a set of affecting changes is determined that mayhave given rise to that test’s changed behavior. Our methodis conservative in the sense that the computed set of affectingchanges is guaranteed to contain at least every change thatmay have caused changes to the test’s behavior.
We will use the example program of Figure 1(a) to il-lustrate our approach. Figure 1(a) depicts two versionsof a simple program comprising classes A, B, and C. Theoriginal version of the program consists of all the programtext except for the 7 program fragments shown in boxes;the edited version of the program consists of all the pro-gram text including the program fragments shown in boxes.Associated with the program are 3 tests, Tests.test1(),Tests.test2(), and Tests.test3().
Our change impact analysis relies on the computation ofa set of atomic changes that capture all source code mod-ifications at a semantic level that is amenable to analysis.We currently use a fairly coarse-grained model of atomicchanges, where changes are categorized as added classes(AC), deleted classes (DC), added methods (AM), deletedmethods (DM), changed methods (CM), added fields (AF),deleted fields (DF), and lookup (i.e., dynamic dispatch)changes (LC)3.
We also compute syntactic dependences between atomicchanges. Intuitively, an atomic change A1 is dependent on
3There are a few more categories of atomic changes that arenot relevant for the example under consideration that willbe presented in Section 3.
another atomic change A2 if applying A1 to the original ver-sion of the program without also applying A2 results in asyntactically invalid program (i.e., A2 is a prerequisite forA1). These dependences can be used to determine that cer-tain changes are guaranteed not to affect a given test, andto construct syntactically valid intermediate versions of theprogram that contain some, but not all atomic changes. It isimportant to understand that the syntactic dependences donot capture semantic dependences between changes (con-sider, e.g., related changes to a variable definition and avariable use in two different methods). This means that iftwo atomic changes, C1 and C2, affect a given test T , thenthe absence of a syntactic dependence between C1 and C2
does not imply the absence of a semantic dependence; thatis, program behaviors resulting from applying C1 alone, C2
alone, or C1 and C2 together, may all be different. If a setS of atomic changes is known to expose a bug, then theknowledge that applying certain subsets of S does not leadto syntactically valid programs, can be used to localize bugsmore quickly.
Figure 1(b) shows the atomic changes that define the twoversions of the example program, numbered 1–13 for conve-nience. Each atomic change is shown as a box, where thetop half of the box shows the category of the atomic change(e.g., CM for changed method), and the bottom half showsthe method or field involved (for LC changes, both the classand method involved are shown). An arrow from an atomicchange A1 to an atomic change A2 indicates that A2 is de-pendent on A1. Consider, for example, the addition of thecall B.bar() in method B.foo(). This source code changeresulted in atomic change 8 in Figure 1(b). Observe thatadding this call would lead to a syntactically invalid programunless method B.bar() is also added. Therefore, atomicchange 8 is dependent on atomic change 6, which is an AMchange for method B.bar(). The observant reader may havenoticed that there is also a CM change for method B.bar()
(atomic change 9). This is the case because our method forderiving atomic changes decomposes the source code changeof adding method B.bar() into two steps: the addition ofan empty method B.bar() (AM atomic change 6 in the fig-ure), and the insertion of the body of method B.bar() (CMatomic change 9 in the figure), where the latter is dependenton the former. Observe that addition of B.bar()’s body re-quires that field B.y be added to class B. Hence, there isa dependence of atomic change 9 on AF atomic change 7,which represents the addition of field B.y. Notice that ourmodel of dependences between atomic changes correctly cap-tures the fact that adding the call to B.bar() to the body ofB.foo() requires that a method B.bar() is added, but notthat field B.y is added.
The LC atomic change category models changes to thedynamic dispatch behavior of instance methods. In particu-lar, an LC change (Y, X.m()) models the fact that a call tomethod X.m() on an object of type Y results in the selectionof a different method. Consider, for example, the addition ofmethod C.foo() to the program of Figure 1(a). As a resultof this change, a call to A.foo() on an object of type C willdispatch to C.foo() in the edited program, whereas it usedto dispatch to A.foo() in the original program. This changein dispatch behavior is captured by atomic change 4. LCchanges are also generated in situations where a dispatchrelationship is added or removed as a result of a source code
change4. For example, atomic changes 5 (defining the be-havior of a call to C.foo() on an object of type C) and 13(defining the behavior of a call to C.baz() on an object oftype C) occur due to the addition of methods C.foo() andC.baz(), respectively.
In order to identify those tests that are affected by a setof atomic changes, we have to construct a call graph for eachtest. The call graphs used in this paper contain one nodefor each method, and edges between nodes to reflect callingrelationships between methods. Our analysis can work withcall graphs that have been constructed using static analysis,or with call graphs that have been obtained by observing theactual execution of the tests. In the experiments reportedin this paper, dynamic call graphs are used.
Figure 1(c) shows the call graphs for the 3 tests test1,test2, and test3, before the changes have been applied.In these call graphs, edges corresponding to dynamic dis-patch are labeled with a pair < T, M >, where T is therun-time type of the receiver object, and M is the methodshown as invoked at the call site. A test is determined to beaffected if its call graph (in the original version of the pro-gram) either contains a node that corresponds to a changedmethod (CM) or deleted method (DM) change, or if itscall graph contains an edge that corresponds to a lookupchange (LC). Using the call graphs in Figure 1(c), it is easyto see that: (i) test1 is not affected, (ii) test2 is affectedbecause its call graph contains a node for B.foo(), whichcorresponds to CM change 8, and (iii) test3 is affectedbecause its call graph contains an edge corresponding to adispatch to method A.foo() on an object of type C, whichcorresponds to LC change 4.
In order to compute the changes that affect a given af-fected test, we need to construct a call graph for that test inthe edited version of the program. These call graphs for thetests are shown in Figure 1(d)5. The set of atomic changesthat affect a given affected test includes: (i) all atomicchanges for added methods (AM) and changed methods(CM) that correspond to a node in the call graph (in theedited program), (ii) atomic changes in the lookup change(LC) category that correspond to an edge in the call graph(in the edited program), and (iii) their transitively prereq-uisite atomic changes.
As an example, we can compute the affecting changes fortest2 as follows. Observe, that the call graph for test2 inthe edited version of the program contains methods B.foo()and B.bar(). These nodes correspond to atomic changes8 and 9 in Figure 1(b), respectively. Atomic change 8 re-quires atomic change 6, and atomic change 9 requires atomicchanges 6 and 7. Therefore, the atomic changes affectingtest2 are 6, 7, 8, and 9. Informally, this means that wecan automatically determine that test2 is affected by theaddition of field B.y, the addition of method B.bar(), andthe change to method B.foo(), but not on any of the othersource code changes! In other words, we can safely rule out 9of the 13 atomic changes as the potential source for test2’schanged behavior.
To conclude our discussion of the example program of Fig-
4Other scenarios that give rise to LC changes will be dis-cussed in Section 3.5The call graph for test1 in the edited version of the pro-gram is not necessary for our analysis because test1 wasnot affected by any of the changes, and is included in thefigure solely for completeness.
class A {public A(){ }public void foo(){ }public int x;
}class B extends A {
public B(){ }public void foo(){ B.bar(); }public static void bar(){ y = 17; }public static int y;
}class C extends A {
public C(){ }public void foo(){ x = 18; }public void baz(){ z = 19; }public int z;
}
class Tests {public static void test1(){
A a = new A();a.foo();
}public static void test2(){
A a = new B();a.foo();
}public static void test3(){
A a = new C();a.foo();
}}
(a)
AF
A.x
1
LC
C,C.foo()
5
LC
C,A.foo()
4CM
C.foo()
2
AM
C.foo()
3
AM
B.bar()
CM
B.foo()
AF
B.y
CM
B.bar()
6 8
7 9
AF
C.z
10
AM
C.baz()
11
CM
C.baz()
12
LC
C,C.baz()
13
(b)
A.A()
Tests.test1()
A.A() A.foo()
Tests.test2()
B.B() B.foo()
A.A()
Tests.test3()
C.C() A.foo()
<A,A.foo()>
<B,A.foo()>
<C,A.foo()>
A.A()
Tests.test1()
A.A() A.foo()
Tests.test2()
B.B()
A.A()
Tests.test3()
C.C() C.foo()
<A,A.foo()>
<B,A.foo()>
B.foo()
B.bar()
<C,A.foo()>
(c) (d)
Figure 1: (a) Example program with 3 tests. Added code fragments are shown in boxes. (b) Atomic changesfor the example program, with their interdependences. (c) Call graphs for the tests before the changes wereapplied. (d) Call graphs for the tests after the changes were applied.
AffectedTests(T ,A) ={ ti | ti ∈ T , Nodes(P, ti) ∩ (CM ∪DM)) 6= ∅ } ∪{ ti | ti ∈ T , n, A.m ∈ Nodes(P, ti),
n→B, X.mA.m ∈ Edges(P, ti),
〈B, X.m〉 ∈ LC, B<∗X }
AffectingChanges(t,A) ={ a′ | a ∈ Nodes(P ′, t) ∩ (CM ∪AM), a′ ¹∗ a } ∪{ a′ | a ≡ 〈B, X.m〉 ∈ LC, B<∗X,
n→B, X.mA.m ∈ Edges(P ′, t),for some n, A.m ∈ Nodes(P ′, t), a′ ¹∗ a }
Figure 2: Affected Tests and Affecting Changes.
ure 1, consider the atomic changes 10, 11, 12, and 13 corre-sponding to the addition of field C.z and method C.baz(),respectively. These atomic changes do not affect any of thetests, indicating that additional tests are needed.
We will use the equations in Figure 2 [21] to more formallydefine how we find affected tests and their corresponding af-fecting atomic changes, in general. Assume the original pro-gram P is edited to yield program P ′, where both P and P ′
are syntactically correct and compilable. Associated with Pis a set of tests T = t1,...,tn. The call graph for test ti on theoriginal program, called Gti , is described by a subset of P ’smethods Nodes(P, ti) and a subset Edges(P, ti) of calling re-lationships between P ’s methods. Likewise, Nodes(P ′, ti)and Edges(P ′, ti) form the call graph G′ti
on the editedprogram P ′. Here, a calling relationship is represented asD.n() →B,X.m() A.m(), indicating possible control flow frommethod D.n() to method A.m() due to a virtual call tomethod X.m() on an object of type B. In these definitions,we implicitly make the usual assumptions [10], namely thatexecution of the program is deterministic and that the li-brary code used and the execution environment (e.g., JVM)itself remain unchanged.
3. ATOMIC CHANGES AND THEIR DEPEN-DENCES
As previously mentioned, a key aspect of our analysis isthe step of uniquely decomposing a source code edit into aset of interdependent atomic changes. In the original formu-lation [21], several kinds of changes, (e.g., changes to accessrights of classes, methods, and fields and addition/deletionof comments) were not modeled. Section 3.1 discusses howthese changes are handled in Chianti. Table 1 lists the set ofatomic changes in Chianti, which includes the original 8 cat-egories [21] plus 8 new atomic changes (the bottom 8 rows ofthe table). Most of the atomic changes are self-explanatoryexcept for CM and LC. CM represents any change to amethod’s body. Some extensions to the original definitionof CM are discussed in detail in Section 3.1. LC representschanges in dynamic dispatch behavior that may be causedby various kinds of source code changes (e.g., by the addi-tion of methods, by the addition or deletion of inheritancerelations, or by changes to the access control modifiers ofmethods). LC is defined as a set of pairs 〈Y, X.m()〉, in-dicating that the dynamic dispatch behavior for a call toX.m() on an object with run-time type Y has changed.
AC Add an empty classDC Delete an empty classAM Add an empty methodDM Delete an empty methodCM Change body of a methodLC Change virtual method lookupAF Add a fieldDF Delete a fieldCFI Change definition of a instance field initializerCSFI Change definition of a static field initializerAI Add an empty instance initializerDI Delete an empty instance initializerCI Change definition of an instance initializerASI Add an empty static initializerDSI Delete an empty static initializerCSI Change definition of an static initializer
Table 1: Categories of atomic changes.
3.1 New and Modified Atomic ChangesChianti handles the full Java programming language, which
necessitated the modeling of several constructs not consid-ered in the original framework [21]. Some of these constructsrequired the definition of new sorts of atomic changes; oth-ers were handled by augmenting the interpretation of atomicchanges already defined.
Initializers, Constructors, and Fields. Six of thenewly added changes in Table 1 correspond to initializers.AI and DI denote the set of added and deleted instanceinitializers respectively, and ASI and DSI denote the set ofadded and deleted static initializers, respectively. CI andCSI capture any change to an instance or static initializer,respectively. The other two new atomic changes, CFI andCSFI, capture any change to an instance or static field, in-cluding (i) adding an initialization to a field, (ii) deleting aninitialization of a field, (iii) making changes to the initializedvalue of a field, and (iv) making changes to a field modifier(e.g., changing a static field into a non-static field).
Changes to initializer blocks and field initializers also haverepercussions for constructors or static initializer methods ofa class. Specifically, if changes are made to initializers of in-stance fields or to instance initializer blocks of a class C, thenthere are two cases: (i) if constructors have been explicitlydefined for class C, then Chianti will report a CM for eachsuch constructor, (ii) otherwise, Chianti will report a changeto the implicitly declared method C.〈init〉 that is generatedby the Java compiler to invoke the superclass’s construc-tor without any arguments. Similarly, the class initializerC.〈clinit〉 is used to represent the method being changedwhen there are changes to a static field (i.e., CSFI) orstatic initializer (i.e., CSI).
Overloading. Overloading poses interesting issues forchange impact analysis. Consider the introduction of anoverloaded method as shown in Figure 3 (the added methodis shown in a box). Note that there are no textual edits inTest.main(), and further, that there are no LC changes be-cause all the methods are static. However, adding methodR.foo(Y) changes the behavior of the program because thecall of R.foo(y) in Test.main() now resolves to R.foo(Y)
instead of R.foo(X) [9]. Therefore, Chianti must report aCM change for method Test.main() despite the fact that
no textual changes occur within this method6.
class R {static void foo(X x){ }static void foo(Y y){ }
}class X { }class Y extends X { }class Test {
static void main(String[] args){Y y = new Y();R.foo(y);
}}
Figure 3: Addition of an overloaded method. Theadded method is shown in a box.
Hierarchy changes. It is also possible for changes to thehierarchy to affect the behavior of a method, although thecode in the method is not changed. Various constructs inJava such as instanceof, casts and exception catch blockstest the run-time type of an object. If such a construct isused within a method and the type lies in a different positionin the hierarchy of the program before the edit and afterthe edit, then the behavior of that method may be affectedby this hierarchy change (or restructuring). For example,in Figure 4(a), method foo() contains a cast to type B.This cast will succeed if the type of the object pointed toby a when execution reaches this statement is B or C inthe original program. In contrast, if we make the hierarchychange shown in Figure 4(b), then this cast will fail if therun-time type of the object which reaches this statement isC. Note that the code in method foo() has not changed dueto the edit, but the behavior of foo() has been possiblyaltered. To capture these sorts of changes in behavior dueto changes in the hierarchy, we report a CM change for themethod containing the construct that checks the run-timetype of the object (i.e., CM(Test.foo())).
class Test {public void foo( ){
A a = new C();...(B)a...
}}
(a)
A A
C
BB C
(b)
Figure 4: Hierarchy change that affects a methodwhose code has not changed.
6However, the abstract syntax tree for Test.main() will bedifferent after applying the edit, as overloading is resolvedat compile time.
Threads and Concurrency. Threads do not pose sig-nificant challenges for our analysis. The addition/deletionof synchronized blocks inside methods and the addi-tion/deletion of synchronized modifiers on methods areboth modeled as CM changes. Threads do not presentsignificant issues for the construction of call graphs either,because the analysis discussed in this paper does not re-quire knowledge about the particular thread that executes amethod. The only information that is required are the meth-ods that have been executed and the calling relationships be-tween them. If dynamic call graphs are used, as is the casein this paper, this information can be captured by tracingthe execution of the tests. If flow-insensitive static analysisis used for constructing call graphs [19], the only significantissue related to threads is to model the implicit calling rela-tionship between Thread.start() and Thread.run().
Exception handling. Exception handling is not a sig-nificant issue in our analysis. Any addition or deletion orstatement-level changes to a try, catch or finally blockwill be reported as a CM change. Similarly, changes to thethrows clause in a method declaration are also captured asCM changes. Possible interprocedural control flow intro-duced by exception handling is expressed implicitly in thecall graph; however, our change impact analysis correctlycaptures effects of these exception-related code changes. Forexample, if a method f() calls a method g(), which in turncalls a method h() and an exception of type E is thrown inh() and caught in g() before the edit, but in f() after theedit, then there will be CM changes for both g() and f()representing the addition and deletion of the correspondingcatch blocks. These CM changes will result in all teststhat execute either f() or g() to be identified as affected.Therefore, all possible effects of this change are taken intoaccount, even without the explicit representation of flow ofcontrol due to exceptions in our call graphs.
Changes to CM and LC. Accommodating methodaccess modifier changes from non-abstract to abstract orvice-versa, and non-public to public or vice-versa, requiredextension of the original definition of CM. CM now com-prises: (i) adding a body to a previously abstract method,(ii) removing the body of a non-abstract method and mak-ing it abstract, or (iii) making any number of statement-level changes inside a method body or any method declara-tion changes (e.g., changing the access modifier from public
to private, adding a synchronized keyword or changing athrows clause).
In addition, in some cases, changing a method’s accessmodifier results in changes to the dynamic dispatch in theprogram (i.e., LC changes). For example, there is no entryfor private or static methods in the dynamic dispatchmap (because they are not dynamically dispatched), but ifa private method is changed into a public method, thenan entry will be added, generating an LC change that isdependent on the access control change, which is representedas a CM. Additions and deletions of import statements mayalso affect dynamic dispatch and are handled by Chianti.
3.2 DependencesAtomic changes have interdependences which induce a
partial ordering ≺ on a set of them, with transitive closure¹∗. Specifically, C1 ¹∗ C2 denotes that C1 is a prerequi-site for C2. This ordering determines a safe order in whichatomic changes can be applied to program P to obtain a syn-
tactically correct edited version P ′′ which, if we apply all thechanges is P ′. Consider that one cannot extend a class Xthat does not yet exist, by adding methods or fields to it (i.e.,AC(X) ≺ AM(X.m()) and AC(X) ≺ AF(X.f)). Thesedependences are intuitive as they involve how new code isadded or deleted in the program. Other dependences aremore subtle. For example, if we add a new method C.m()and then add a call to C.m() in method D.n(), there will be adependence AM(C.m()) ≺ CM(D.n()). Figure 1(b) showssome examples of dependences among atomic changes.
Dependences involving LC changes can be caused by ed-its that alter inheritance relations. LC changes can beclassified as (i) newly added dynamic dispatch tuples (e.g.,caused by declaring a new class/interface or method), (ii)deleted dynamic dispatch tuples (e.g., caused by deletinga class/interface or method), or (iii) dynamic dispatch tu-ples with changed targets (e.g., caused by adding/deleting amethod or changing the access control of a class or method).For example, making an abstract class C non-abstract willresult in LC changes. In the original dynamic dispatch map,there is no entry with C as the run-time receiver type, butthe new dispatch map will contain such an entry. Similardependences result when other access modifiers are changed.
3.3 Engineering IssuesOne engineering problem encountered in building Chianti
resulted from the absence of unique names for anonymousand local classes. In a JVM, anonymous classes are repre-sented as EnclosingClassName$〈num〉, where the numberassigned represents the lexical order of the anonymous classwithin its enclosing class. This naming strategy guaran-tees that all the class names in a Java program are unique.However, when Chianti compares and analyzes two relatedJava programs, it needs to establish a correspondence be-tween classes and methods in each version to determine theset of atomic changes. The approach used is a match-by-name strategy in which two components in different pro-grams match if they have the same name; however, whenthere are changes to anonymous or local inner classes, thisstrategy requires further consideration.
Figure 5 shows a simple program using anonymousclasses with the code added by the edit shown inside abox. In this program, method listJavaFiles(String) listsall Java files in a directory that is specified by its pa-rameter. Anonymous class Lister$1 implements interfacejava.io.FilenameFilter and is defined as part of a methodcall expression. Now, assume that the program is edited anda method listClassFiles(String) is added that lists allclass files in a directory. This new method declares another,similar anonymous class. Now, in the edited version of theprogram, the Java compiler will name this new anonymousclass Lister$1 and the previous anonymous class, formerlynamed Lister$1, will become Lister$2. Clearly, the match-by-name strategy cannot be based on compiler-generatednames because the original anonymous class has differentnames before and after the edit.
To solve this problem, Chianti uses a naming strategyfor classes that assigns each a unique internal name.For top-level classes or member classes, the internalname is the same as the class name. For anonymousclasses and local inner classes, the unique name consistsof four parts: enclosingClassName, enclosingElement-Name, selfSuperclassInterfacesName, sequenceNumber.
import java.io.*;class Lister {
static void listClassFiles(String dir){File f = new File(dir);String[] list = f.list(
new FilenameFilter() { //anonymous classboolean accept(File f, String s){
return s.endsWith(".class");}
});for(int i = 0; i < list.length; i++)
System.out.println(list[i]);}static void listJavaFiles(String dir){
File f = new File(dir);String[] list = f.list(new FilenameFilter() { //anonymous class
boolean accept(File f,String s){return s.endsWith(".java");
}});
for(int i = 0; i < list.length; i++)System.out.println(list[i]);
}}
Figure 5: Addition of an anonymous class. Theadded code fragments are shown inside a box.
For the example in Figure 5, the unique internalname of the anonymous class in the original programis Lister$listJavaF iles(String)$java.io.F ilenameFilter$1,while the unique internal name of the newlyadded anonymous class in the edited program isLister$listClassF iles(String)$java.io.F ilenameFilter$1.Similarly, the internal name of the origi-nal anonymous class in the edited program isLister$listJavaF iles(String)$java.io.F ilenameFilter$1.Notice that this original anonymous class whose compiler-generated names are Lister$1 in the original programand Lister$2 in the edited program, has the same uniqueinternal name in both versions. With this new namingstrategy, match-by-name can identify anonymous and localinner classes and report atomic changes involving them7.
4. PROTOTYPEChianti has been implemented in the context of the Java
editor of Eclipse, a widely used extensible open-source de-velopment environment for Java. Our tool is designed as acombination of Eclipse views, a plugin, and a launch con-figuration that together constitute a change impact analysisperspective8. Chianti is built as a plugin of Eclipse and con-ceptually can be divided into three functional parts. Onepart is responsible for deriving a set of atomic changes fromtwo versions of an Eclipse project (i.e., Java program), whichis achieved via a pairwise comparison of the abstract syn-
7This naming scheme can only fail when two anonymousclasses occur within the same scope and extend the samesuperclass. If this occurs due to an edit, however, Chiantigenerates a safe set of atomic changes corresponding to theedit.8A perspective is Eclipse terminology for a collection ofviews that support a specific task, (e.g., the Java perspectiveis used for creating Java applications).
CHIANTI
Original Program P Changed Program P’Set of Unit Tests
Atomic Change
Decoder
Atomic Changes
& Dependences
Call Graph Builder
Call Graphs of Tests in
P
Call Graphs of Tests
in P’
Change Impact
AnalyzerAffected Tests Affecting Changes
Figure 6: Chianti architecture.
tax trees of the classes9 in the two project versions. An-other part reads test call graphs for the original and editedprojects, computes affected tests and their affecting changes.The third part manages the views that allow the user tovisualize change impact information. Chianti’s GUI also in-cludes a launch configuration that allows users to select theproject versions to be analyzed, the set of tests associatedwith the project and the call graphs to be used. Figure 6depicts Chianti’s architecture.
Although Chianti is intended for interactive use, we havebeen testing the prototype using successive CVS versionsof a program. Thus, a typical scenario of a Chianti ses-sion begins with the programmer extracting two versions ofa project from a CVS version control repository into theworkspace. The programmer then starts the change impactanalysis launch configuration, and selects the two projectsof interest as well as the test suite associated with theseprojects. Currently, we allow tests that have a separatemain() routine and JUnit tests10.
In order to enable the reuse of analysis results, and todecouple the analysis from GUI-related tasks, both atomicchange information and call graphs are stored as XML files.Chianti currently supports two mechanisms for obtainingthe call graphs to be used in the analysis. When static callgraphs are desired, Chianti invokes the Gnosis analysis en-gine11 to construct these [19]. In this case, users need tosupply some additional information relevant to the analysisengine (e.g., the choice of call graph construction algorithmto be used and some policy settings for dealing with reflec-tion).
Users can also point Chianti directly at an XML file rep-resentation of the call graphs that are to be used, in order toenable the use of call graphs that have been constructed byexternal tools. The experiments with dynamic call graphs
9While Eclipse provides functionality for comparing sourcefiles at a textual level, we found the amount of informa-tion provided inadequate for our purposes. In particular,the class hierarchy information provided by Eclipse does notcurrently include anonymous and local classes.
10See www.junit.org.11Gnosis is a static analysis framework that has been devel-oped at IBM Research as a test-bed for research on demand-driven and context-sensitive static analysis.
presented in this paper have been conducted using an off-line tool that instruments the class files for an application.Executing an application that has been instrumented by thistool produces an XML file containing the application’s dy-namic call graph12.
When the analysis results are available, the Eclipse work-bench changes to the change impact analysis perspective,which provides a number of views:
• The affecting changes view shows all tests in a treeview. Each affected test can be expanded to show itsset of affecting changes and their prerequisites. Fig-ure 7 shows a snapshot of this view; note how theprerequisite changes are shown. Each atomic changeis the root of a tree that can be expanded on demandto show prerequisite changes. This quickly providesan idea of the different “threads’’ of changes that haveoccurred.
• The atomic-changes-by-category view shows the differ-ent atomic changes grouped by category.
Each of these user interface components is seamlessly inte-grated with the standard Java editor in Eclipse (e.g., clickingon an atomic change in the affecting changes view opens aneditor on the associated program fragment).
5. EVALUATIONThe experiments with Chianti were performed on versions
of the Daikon system by M. Ernst et al. [7], extracted fromthe developers’ CVS repository. The Daikon CVS repositorydoes not use version tags, so we partitioned the year-longversion history arbitrarily at week boundaries. All modi-fications checked in within a week were considered to bewithin one edit whose impact was to be determined. How-ever, in cases where no editing activity took place in a givenweek, we extended the interval by one week until it includedchanges. The data reported in this section covers the entireyear 2002 (i.e., 52 weeks) of updates, during which therewere 39 intervals with editing activity.
During the year under consideration, Daikon was activelybeing developed and increased in size from 48K to 123K linesof code. More significant are the program-based measuresof growth, from 357 to 755 classes, 2878 to 7112 methods,and 937 to 2885 fields. The number of unit tests associatedwith Daikon grew from 40 to 62 during the time periodunder consideration. Figure 8 shows in detail the growthcurves over this time period. Clearly, this is a moderate-sized application that experienced considerable growth insize (and complexity) over the year 2002.
5.1 Atomic ChangesFigure 9(a) shows the number of atomic changes between
each pair of versions. The number of atomic changes perinterval varies greatly between 1 and 11,698 during this pe-riod, although only 11 edits involved more than 1,000 atomicchanges. Section 5.3 gives more details about two specificintervals in our study.
12We did not optimize the gathering of the dynamic call in-formation; presently, the instrumented tests run, on aver-age, about 2 orders of magnitude more slowly than unin-strumented code, but we think we can reduce this overheadsignificantly with some effort.
Figure 7: Snapshot of Chianti’s affecting changes view. The arrow shows how clicking on an atomic changeopens an editor on the associated source fragment.
Figure 9(b) summarizes the relative percentages of kindsof atomic changes observed during 2002. The height of eachbar indicates the frequency of the corresponding kind ofatomic change; these values vary widely, by three ordersof magnitude. Three of our atomic change categories werenot seen in this data, namely addInitializer, changeInitial-izer and deleteInitializer13. Note that the 0.01% value fordeleteStaticInitializer in the figure represents the 5 atomicchanges of that type out of a total of over 44,000 changesfor the entire year!
Figure 10 shows the proportion of atomic changes per in-terval, grouped by the program construct they affect, namely,classes, fields, methods and dynamic dispatch. Clearly, thetwo most frequent groups of atomic changes are changes todynamic dispatch (i.e., LC) and changes to methods (i.e.,CM); their relative amounts vary over the period.
5.2 Affected Tests and Affecting ChangesFigure 11 shows the percentage of affected tests for each
of the Daikon versions. On average, 52% of the tests are af-fected in each edit. Interestingly, there were several intervalsover which no tests were affected, although atomic changesdid occur. For example, there were no affected tests for the
13This is not surprising because, in Java, instance initializersare only needed in the rare event that an anonymous classneeds to perform initialization actions that cannot be ex-pressed using field initializers. In non-anonymous classes, itis generally preferable to incorporate initialization code inconstructors or in field initializers.
interval between 04/01/02 and 04/08/02, despite the factthat there were 212 atomic changes during this time. Sim-ilarly, for the interval between 8/26/02 and 9/02/02 therewere 286 atomic changes, but no affected tests. This meansthat the changed code for these intervals was not coveredby any of the tests! In principle, a change impact analysistool could inform the user that additional unit tests shouldbe written when an observation of this kind is made.
Figure 12 shows the average percentage of affecting changesper affected test, for each of the Daikon versions. On av-erage, only 3.95% of the atomic changes impact a given af-fected test. This means that our technique has the potentialof dramatically reducing the amount of time required for de-bugging when a test produces an erroneous result after anediting session.
By contrast, an earlier study performed with Chianti us-ing static call graphs for the same Daikon data, yielded onaverage 56% affected tests and 3.7% affecting changes peraffected test [19]14. The closeness of these results to thosereported in the present paper suggests that we should inves-tigate the tradeoffs associated with using static or dynamiccall graphs.
14Imprecision in the static call graphs resulted in the detec-tion of extra affected tests that had relatively small numbersof affecting changes. This skewed our averaging calculationsto yield the counterintuitive result that the affecting changespercentage obtained using static call graphs was lower thanthe percentage obtained using the more precise dynamic callgraphs.
30
40
50
60
70
80
90
100
110
120
130
1/7
1/21 2/4
2/18 3/4
3/18 4/1
4/15
4/29
5/13
5/27
6/10
6/25 7/8
7/22 8/5
8/19 9/2
9/16
9/30
10/1
4
10/2
8
11/1
1
11/2
6
12/9
12/2
3
Date
# o
f K
LO
C, U
nit
Tes
ts
0
1000
2000
3000
4000
5000
6000
7000
8000
# o
f M
eth
od
s, F
ield
s, C
lass
es
# of KLOC
# of Unit Tests
# of Methods
# of Fields
# of Classes
Figure 8: Daikon growth statistics for the year 2002
Our approach assumes that the test suite associated witha Java program offers good coverage of the entire program.To verify this assumption, we used the JCoverage tool (seewww.jcoverage.com) to determine how many methods inDaikon were actually exercised by its unit test suite. Foreach version of Daikon, we obtained the number of meth-ods covered by the associated tests and the total number of(source code) methods in that version, yielding an averagemethod coverage ratio. The overall average of these ratioson the entire Daikon system is quite low, at 21%. How-ever, this number is skewed by the fact that certain Daikoncomponents have reasonable coverage (e.g., for the utilMDE
component we find an average coverage ratio over the yearof 47%), whereas other components (e.g., the jtb compo-nent) have virtually no coverage. Thus, while our changeimpact analysis findings are promising, they would be morecompelling with a test suite offering better coverage of thesystem.
5.3 Case Studies
We conducted two detailed case studies to further inves-tigate the possible applications of Chianti as it is intendedto be used, namely in interactive environments with shorttime intervals between versions. To this end, we selectedtwo one-week intervals from the whole year’s data in whichheavy editing activity occurred, and divided those intervalsinto subintervals of one day each.
Case Study 1 The first interval we decided to explorefurther is the one for which we found the highest percentageof affected tests. This occurred between versions 07/08/02and 07/15/02, when 88.7% (55 out of 62) of the tests wereaffected. We partitioned the version history of this intervalinto daily intervals so that we could obtain changes with
finer granularity. In cases where no editing activity tookplace between two days, we extended the interval by oneday, thus obtaining 5 intervals with editing activity.
Figure 13(a) shows the number of affected tests for eachsubinterval as well as the number of affected tests for theoriginal week-long interval (shown as the rightmost pair ofbars). Before partitioning, 55 of the 62 unit tests were af-fected tests, but smaller numbers of affected tests, rangingfrom 1 to 53, were reported for each of the subintervals (forexample, in subinterval 07/10/02—07/11/02, there is onlyone affected test).
Figure 13(b) shows the total number of atomic changesand the average number of affecting changes per affectedtest in each subinterval compared with the original inter-val (again shown as the rightmost pair of bars). The useof smaller intervals resulted in smaller numbers of atomicchanges for each interval and also smaller numbers of affect-ing changes per affected test; this makes the tracing of af-fecting changes much easier. In addition, we found that 12 ofthe 55 affected tests for the original, week-long interval wereonly affected in one of the smaller intervals, which meansthat we can narrow down the range of affecting changes intoa small set of atomic changes for these 12 tests.
Case Study 2 The second interval we selected is the onewith the highest average number of affecting changes. Thisinterval took place between versions 01/21/02 and 01/28/02,when 140 affecting changes occurred on average (rangingfrom 3 to 217) for 32 affected tests. Similar to case study1, we partitioned the original week-long interval into severalsubintervals, obtaining 3 subintervals with editing activity.In Figure 14(a) and (b) we can see similar results to those ofcase study 1, that is, we obtain smaller numbers of atomicchanges, affected tests and affecting changes compared tothe original week interval.
268
1471
2810
308
11698
1197
171
397
1006
212
116
300
29
214
350
5238
378
2344
1319
4
286
3
15
65
7
13
6095
153
1
163
635
40
2913
659
723
665
332
465
1747
1
10
100
1000
10000
100000
0107-0114
0114-0121
0121-0128
0128-0204
0204-0211
0211-0218
0218-0225
0225-0304
0304-0311
0311-0318
0318-0401
0401-0408
0408-0415
0415-0506
0506-0527
0527-0603
0603-0610
0610-0617
0617-0625
0625-0701
0701-0708
0708-0715
0715-0722
0722-0805
0805-0819
0819-0826
0826-0902
0902-0909
0909-0916
0916-0923
0923-0930
0930-1111
1111-1119
1119-1126
1126-1202
1202-1209
1209-1216
1216-1223
1223-1230
(a)
1.05% 0.16%
12.36%
19.28%
2.19%4.90%
0.73%2.77%
0.55% 0.05% 0.14% 0.01%
55.82%
0%
10%
20%
30%
40%
50%
60%
addClass
deleteClass
addMethod
changeMethod
deleteMethod
addField
changeFieldInitializer
changeStaticFieldInitializer
deleteField
addStaticInitializer
changeStaticInitializer
deleteStaticInitializer
lookupChange
(b)
Figure 9: (a) Number of atomic changes between each pair of Daikon versions in 2002 (note the log scale).(b) Categorization of the atomic changes, aggregated over all Daikon edits in 2002.
0%
20%
40%
60%
80%
100%
0107-0
114
0114-0
121
0121-0
128
0128-0
204
0204-0
211
0211-0
218
0218-0
225
0225-0
304
0304-0
311
0311-0
318
0318-0
401
0401-0
408
0408-0
415
0415-0
506
0506-0
527
0527-0
603
0603-0
610
0610-0
617
0617-0
625
0625-0
701
0701-0
708
0708-0
715
0715-0
722
0722-0
805
0805-0
819
0819-0
826
0826-0
902
0902-0
909
0909-0
916
0916-0
923
0923-0
930
0930-1
111
1111-1
119
1119-1
126
1126-1
202
1202-1
209
1209-1
216
1216-1
223
1223-1
230
lookup changes class changes method changes field changes
Figure 10: Classification of atomic changes for each pair of versions. Class changes include AC and DC.Method changes include AM, CM, DM, ASI, DSI and CSI. Field changes include AF, DF, CSFI and CFI.
In both case studies, we found that the use of subin-tervals with smaller numbers of affecting changes improvesthe ability of Chianti to help programmers with under-standing the effects of an edit. Even in subintervalssuch as 01/21/02—01/25/02, where the number of atomicchanges and the average number of affecting changes arelarge relative to the corresponding numbers for the origi-nal interval, Chianti can provide useful insights. For ex-ample, consider one test with a large number of affect-ing changes: daikon.test.diff.DiffTester.testPpt4Ppt4 fromsubinterval 01/21/02—01/25/02. The affecting changes forthis test are: 67 CM changes, 67 AF changes, and 69 CSFIchanges. Among the 67 CM changes, 65 of them are asso-ciated with static initializers for some class. These, in turn,are dependent on 68 of the CSFIs, whose own prerequisitesare 66 of the AFs. A closer look revealed that all the addedfields have the same name, serialVersionUID, which is usedto add serialization-related functionality to Daikon. It is in-teresting to observe that Chianti was able to determine thatthe changed behavior of this test was almost entirely due tothis serialization-related change, and that the other 800+atomic changes that occurred during this interval did notcontribute to the test’s changed behavior.
5.4 Chianti PerformanceThe performance of Chianti has thus far not been our
primary focus, however, we have achieved acceptable per-formance for a prototype. Deriving atomic changes fromtwo successive versions of Daikon takes, on average, approx-
imately 87 seconds. Computing the set of affected tests foreach version pair takes approximately 5 seconds on average,and computing affecting changes takes on average approxi-mately 1.2 seconds per affected test. All measurements weretaken on a Pentium 4 PC at 2.8Ghz with 1Gb RAM.
6. RELATED WORKIn previous papers, we presented the conceptual frame-
work of our change impact analysis, without empirical ex-perimentation [21], and reported on experiments with a purelystatic version of Chianti using static call graphs generatedby Gnosis, on the same Daikon data used here [19].
We distinguish three broad categories of related work inthe community: (i) change impact analysis techniques, (ii)regression test selection techniques, and (iii) techniques forcontrolling the way changes are made.
6.1 Change Impact Analysis TechniquesPrevious research in change impact analysis has varied
from approaches relying completely on static information,including the early analyses of [3, 11], to approaches thatonly utilize dynamic information, such as [13]. There alsoare some methods [15] that use a combination of static anddynamic information. The method described in this paperis a combined approach, in that it uses (i) static analysis forfinding the set of atomic changes comprising a program editand (ii) dynamic call graphs to find the affected tests andtheir affecting changes.
53%
80%
0%2%
63%
89%
0%
68%
0%
61%
70%
0%
61%
52%
22%
27%
68%65%
60%
44%
48%
76%
57%
0%
69%
77%
0%
63%
76%
63%
79%
68%
72%72%
63%
67%69%
57%
55%
58%
0.0%
20.0%
40.0%
60.0%
80.0%
100.0%
0107_0114
0114_0121
0121_0128
0128_0204
0204_0211
0211_0218
0218_0225
0225_0304
0304_0311
0311_0318
0318--0401
0401--0408
0408--0415
0415--0506
0506--0527
0527--0603
0603--0610
0610--0617
0617--0625
0625--0701
0701-0708
0708-0715
0715-0722
0722-0805
0805-0819
0819-0826
0826-0902
0902-0909
0909-0916
0916-0923
0923-0930
0930-1111
1111-1119
1119-1126
1126-1202
1202-1209
1209-1216
1216-1223
1223-1230
Average
Percentage Affected Tests
Figure 11: Percentage of affected tests for each of the Daikon versions.
9.54%
0.34%
2.58%
15.07%
10.34%
28.24%
2.94%
5.08%
73.53%
43.41%
3.82% 3.95%
0.38%
0.26%
8.33%
5.64%
3.60%
2.88%
1.95%
7.68%
8.81%
1.54%
3.84%
1.44%
12.67%
3.36%
5.90%
0.48%
1.96% 1.92%
33.16%
1.21%
1.29%
5.94%
0.10%
1.00%
10.00%
100.00%
0107_0114
0114_0121
0121_0128
0128_0204
0204_0211
0211_0218
0218_0225
0225_0304
0304_0311
0311_0318
0318--0401
0408--0415
0415--0506
0506--0527
0527--0603
0603--0610
0610--0617
0617--0625
0625--0701
0701-0708
0708-0715
0715-0722
0722-0805
0805-0819
0819-0826
0909-0916
0916-0923
0930-1111
1111-1119
1119-1126
1202-1209
1209-1216
1223-1230
Geom-mean
Percentage Affecting Changes
Figure 12: Average percentage of affecting changes, per affected test, for each of the Daikon versions.
62 62 62 62 62 62
42
37
1
53
41
55
0
10
20
30
40
50
60
70
0708_0709 0709_0710 0710_0711 0711_0712 0712_0715 0708-0715
Total tests Affected Tests
(a) Number of affected tests on large and smallerdaily intervals.
2515
87
327
4834 5238
7
2
7
15
64 68
1
10
100
1000
10000
0708_0709 0709_0710 0710_0711 0711_0712 0712_0715 0708_0715
Total Atomic Changes Affecting Changes on Average
(b)Number of atomic changes and on average af-fecting changes on large interval and smaller dailyintervals (note log scale)
Figure 13: Detailed analysis results for the interval 7/08/02—7/15/02.
42 42 42 42
32
1
22
32
0
5
10
15
20
25
30
35
40
45
0121_0125 0125_0127 0127_0128 0121-0128
Total tests Affected Tests
(a) Number of affected tests on large and smallerdaily intervals.
1046
5
463
1471
132
3
12
140
1
10
100
1000
10000
0121_0125 0125_0127 0127_0128 0121-0128
Total Atomic Changes Affecting Changes on Average
(b)Number of atomic changes and on average af-fecting changes on large interval and smaller dailyintervals (note log scale)
Figure 14: Detailed analysis results for the interval 1/21/02—1/28/02.
All of the impact analyses previous to ours focus on find-ing constructs of the program potentially affected by codechanges. In contrast, our change impact analysis aims tofind a subset of the changes that impact a test whose be-havior has (potentially) changed. First we will discuss theprevious static techniques and then address the combinedand dynamic approaches.
An early form of change impact analysis used reachabilityon a call graph to measure impact. This technique15 waspresented by Bohner and Arnold [3] as “intuitively appeal-ing” and “a starting point” for implementing change impactanalysis tools. However, applying the Bohner-Arnold tech-nique is not only imprecise but also unsound, because, bytracking only methods downstream from a changed method,it disregards callers of that changed method that can alsobe affected.
Kung et al. [11] described various sorts of relationships be-tween classes in an object relation diagram (i.e., ORD), clas-sified types of changes that can occur in an object-orientedprogram, and presented a technique for determining changeimpact using the transitive closure of these relationships.
15This is only one of the static change impact analyses dis-cussed.
Some of our atomic change types partially overlap with theirclass changes and class library changes.
More recently, Tonella’s impact analysis [27] determines ifthe computation performed on a variable x affects the com-putation on another variable y using a number of straightfor-ward queries on a concept lattice that models the inclusionrelationships between a program’s decomposition (static)slices [8]. Tonella reports some metrics of the computedlattices, but gives no assessment of the usefulness of his tech-niques.
A number of tools in the Year 2000 analysis domain [5, 18]use type inference to determine the impact of a restricted setof changes (e.g., expanding the size of a date field) and per-form them if they can be shown to be semantics-preserving.
Thione et al. [25, 24] wish to find possible semantic inter-ferences introduced by concurrent programmer insertions,deletions or modifications to code maintained with a versioncontrol system. In this work, a semantic interference is char-acterized as a change that breaks a def-use relation. Theirunit of program change is a delta provided by the versioncontrol system, with no notion of subdividing this delta intosmaller units, such as our atomic changes. Their analysis,which uses program slicing, is performed at the statementlevel, not at the method level as in Chianti. No empirical
experience with the algorithm is given.The CoverageImpact change impact analysis technique by
Orso et al. [15] uses a combined methodology, by correlatinga forward static slice [26] with respect to a changed programentity (i.e., a basic block or method) with execution data ob-tained from instrumented applications. Each program en-tity change is thusly associated with a set of possibly affectedprogram entities. Finally, these sets are unioned to form thefull change impact set corresponding to the program edit.
There are a number of important differences between ourwork and that by Orso et al. First, we differ in the goals ofthe analysis. The method of Orso et al. [15] is focused onfinding those program entities that are possibly affected by aprogram edit. In contrast, our method is focused on findingthose changes that caused the behavioral differences in atest whose behavior has changed. Second, the granularityof change expressed in their technique is a program entity,which can vary from a basic block to an entire method. Incontrast, we use a richer domain of changes more familiar tothe programmer, by taking a program edit and decomposingit into interdependent, atomic changes identified with thesource code (e.g., add a class, delete a method, add a field).Third, their technique is aimed at deployed codes, in thatthey are interested in obtaining user patterns of programexecution. In contrast, our techniques are intended for useduring the earlier stages of software development, to givedevelopers immediate feedback on changes they make.
Law and Rothermel [13] present PathImpact, a dynamicimpact analysis that is based on whole-path profiling [12].In this approach, if a procedure p is changed, any procedurethat is called after p, as well as any procedure that is on thecall stack after p returns, is included in the set of potentiallyimpacted procedures. Although our analysis differs fromthat of Law and Rothermel in its goals (i.e., finding affectedprogram entities versus finding changes affecting tests), bothanalyses use the same method-level granularity to describechange impact.
A recent empirical comparison [16] of the dynamic impactanalyses CoverageImpact by Orso et al. [15] and PathImpactby Law and Rothermel [13] revealed that the latter computesmore precise impact sets than the former in many cases, butuses considerably (7 to 30 times) more space to store exe-cution data. Based on the reported performance results,the practicality of PathImpact on programs that generatelarge execution traces seems doubtful, whereas CoverageIm-pact [16] does appear to be practical, although it can besignificantly less precise. Another outcome of the study isthat the relative difference in precision between the two tech-niques varies considerably across (versions of) programs, andalso depends strongly on the locations of the changes.
Zeller [28] introduced the delta debugging approach for lo-calizing failure-inducing changes among large sets of textualchanges. Efficient binary-search-like techniques are used topartition changes into subsets, executing the programs re-sulting from applying these subsets, and determining whetherthe result is correct, incorrect, or inconclusive. An impor-tant difference with our work is that our atomic changesand interdependences take into account program structureand dependences between changes, whereas Zeller assumesall changes to be completely independent.
6.2 Regression Test SelectionSelective regression testing16 aims at reducing the number
of regression tests that must be executed after a softwarechange [20, 17]. These techniques typically determine theentities in user code that are covered by a given test, andcorrelate these against those that have undergone modifica-tion, to determine a minimal set of tests that are affected.
Several notions of coverage have been used. For example,TestTube [4] uses a notion of module-level coverage, and De-jaVu [20] uses a notion of statement-level coverage. The em-phasis in this work is mostly on reducing the cost of runningregression tests, whereas our interest is primarily in assist-ing programmers with understanding the impact of programedits.
Bates and Horwitz [1] and Binkley [2] proposed fine-grainednotions of program coverage based on program dependencegraphs and program slices, with the goal of providing as-sistance with understanding the effects of program changes.In comparison to our work, this work uses more costly staticanalyses based on (interprocedural) program slicing and con-siders program changes at a lower-level of granularity, (e.g.,changes in individual program statements).
Our technique for change impact analysis uses affectedtests to indicate to the user the functionality that has beenaffected by a program edit. Our analysis determines a sub-set of those tests associated with a program which need tobe rerun, but it does so in a very different manner than pre-vious selective regression testing approaches, because the setof affected tests is determined without needing informationabout test execution on both versions of the program.
Rothermel and Harrold [20] present a regression test se-lection technique that relies on a simultaneous traversal oftwo program representations (control flow graphs (CFGs)in [20]) to identify those program entities (edges in [20])that represent differences in program behavior. The tech-nique then selects any modification-traversing test that istraversing at least one such “dangerous” entity. This regres-sion test selection technique is safe in the sense that any testthat may expose faults is guaranteed to be selected.
Harrold et al. [10] present a safe regression test selectiontechnique for Java that is an adaptation of the techniqueof Rothermel and Harrold [20]. In this work, Java Inter-class Graphs (JIGs) are used instead of control-flow graphs.JIGs extend CFGs in several respects: Type and class hi-erarchy information is encoded in the names of declarationnodes, a model of external (unanalyzed) code is used for in-complete applications, calling relationships between meth-ods are modeled using Class Hierarchy Analysis, and addi-tional nodes and edges are used for the modeling of exceptionhandling constructs.
The method for finding affected tests presented in this pa-per is also safe in the sense that it is guaranteed to identifyany test that reveals a fault. However, unlike the regres-sion test selection techniques such as [20, 10], our methoddoes not rely on a simultaneous traversal of two representa-tions of the program to find semantic differences. Instead,we determine affected tests by first deriving from a sourcecode edit a set of atomic changes, and then correlating thosechanges with the nodes and edges in the call graphs for the
16 We use the term broadly here to indicate any methodologythat tries to reduce the time needed for regression testingafter a program change, without missing any test that maybe affected by that change.
tests in the original version of the program. Investigatingthe cost/precision tradeoffs between these two approachesfor finding tests that are affected by a set of changes is atopic for further research.
In the work by Elbaum et al. [6], a large suite of regres-sion tests is assumed to be available, and the objective isto select a subset of tests that meets certain (e.g., cover-age) criteria, as well as an order in which to run these teststhat maximizes the rate of fault detection. The differencebetween two versions is used to determine the selection oftests, but unlike our work, the techniques are to a large ex-tent heuristics-based, and may result in missing tests thatexpose faults.
The change impact analysis of [15] can be used to providea method for selecting a subset of regression tests to bererun. First, all the tests that execute the changed programentities are selected. Then, there is a check if the selectedtests are adequate for those program changes. Intuitively, anadequate test set T implies that every relationship between aprogram entity change and a corresponding affected entity istested by a test in T . In their approach, they can determinewhich affected entities are not tested (if any). Accordingto the authors, this is not a safe selective regression testingtechnique, but it can be used by developers, for example, toprioritize test cases and for test suite augmentation.
6.3 Controlling the Change ProcessPalantir [22] is a tool that informs users of a configura-
tion management system when other users access the samemodules and potentially create direct conflicts.
Lucas et al [23] describes reuse contracts, a formalism toencapsulate design decisions made when constructing an ex-tensible class hierarchy. Problems in reuse are avoided bychecking proposed changes for consistency with a specifiedset of possible operations on reuse contracts.
7. CONCLUSIONS AND FUTURE WORKWe have presented our experiences with Chianti, a change
impact analysis tool that has been validated on a year ofCVS data from Daikon. Our empirical results show thatafter a program edit, on average the set of affected tests isa bit more than half of all the possible tests (52%) and foreach affected test, the number of affecting changes is verysmall (3.95% of all atomic changes in that edit). These find-ings suggest that our change impact analysis is a promisingtechnique for both program understanding and debugging.
Plans for future research include an in-depth evaluationof the cost/precision tradeoffs involved in using static ver-sus dynamic call graphs, now that we have some experiencewith both. We also intend to experiment with smaller unitsof change, to better describe change impact to a user, espe-cially since we currently consider all changes to code withina method (i.e., CM) as one monolithic change.
Acknowledgements. We would like to thank MichaelErnst and his research group at MIT for the use of theirdata. We are also grateful to the anonymous reviewers fortheir constructive feedback.
8. REFERENCES[1] Bates, S., and Horwitz, S. Incremental program
testing using program dependence graphs. In Proc. ofthe ACM SIGPLAN-SIGACT Conf. on Principles of
Programming Languages (POPL’93) (Charleston, SC,1993), pp. 384–396.
[2] Binkley, D. Semantics guided regression test costreduction. IEEE Trans. on Software Engineering 23, 8(August 1997).
[3] Bohner, S. A., and Arnold, R. S. An introductionto software change impact analysis. In SoftwareChange Impact Analysis, S. A. Bohner and R. S.Arnold, Eds. IEEE Computer Society Press, 1996,pp. 1–26.
[4] Chen, Y., Rosenblum, D., and Vo, K. Testtube: Asystem for selective regression testing. In Proc. of the16th Int. Conf. on Software Engineering (1994),pp. 211–220.
[5] Eidorff, P. H., Henglein, F., Mossin, C., Niss,H., Sorensen, M. H., and Tofte, M. AnnoDomini:From type theory to year 2000 conversion. In Proc. ofthe ACM SIGPLAN-SIGACT Symp. on Principles ofProgramming Languages (January 1999), pp. 11–14.
[6] Elbaum, S., Kallakuri, P., Malishevsky, A. G.,Rothermel, G., and Kanduri, S. Understandingthe effects of changes on the cost-effectiveness ofregression testing techniques. Journal of SoftwareTesting, Verification, and Reliability (2003). Toappear.
[7] Ernst, M. D. Dynamically discovering likely programinvariants. PhD thesis, University of Washington,2000.
[8] Gallagher, K., and Lyle, J. R. Using programslicing in software maintenance. IEEE Trans. onSoftware Engineering 17 (1991).
[9] Gosling, J., Joy, B., Steele, G., and Bracha, G.The Java Language Specification (Second Edition).Addison-Wesley, 2000.
[10] Harrold, M. J., Jones, J. A., Li, T., Liang, D.,Orso, A., Pennings, M., Sinha, S., Spoon, S. A.,and Gujarathi, A. Regression test selection for Javasoftware. In Proc. of the ACM SIGPLAN Conf. onObject Oriented Programming Languages and Systems(OOPSLA’01) (October 2001), pp. 312–326.
[11] Kung, D. C., Gao, J., Hsia, P., Wen, F.,Toyoshima, Y., and Chen, C. Change impactidentification in object oriented software maintenance.In Proc. of the International Conf. on SoftwareMaintenance (1994), pp. 202–211.
[12] Larus, J. Whole program paths. In Proc. of the ACMSIGPLAN Conf. on Programming Language Designand Implementation (May 1999), pp. 1–11.
[13] Law, J., and Rothermel, G. Whole programpath-based dynamic impact analysis. In Proc. of theInternational Conf. on Software Engineering (2003),pp. 308–318.
[14] Milanova, A., Rountev, A., and Ryder, B. G.Precise call graphs for C programs with functionpointers. Journal for Automated Software Engineering(2004). Special issue on Source Code Analysis andManipulation.
[15] Orso, A., Apiwattanapong, T., and Harrold,M. J. Leveraging field data for impact analysis andregression testing. In Proc. of European SoftwareEngineering Conf. and ACM SIGSOFT Symp. on theFoundations of Software Engineering (ESEC/FSE’03)
(Helsinki, Finland, September 2003).
[16] Orso, A., Apiwattanapong, T., Law, J.,Rothermel, G., and Harrold, M. J. An empiricalcomparison of dynamic impact analysis algorithms. InProc. of the International Conf. on SoftwareEngineering (ICSE’04) (Edinburgh, Scotland, 2004),pp. 491–500.
[17] Orso, A., Shi, N., and Harrold, M. J. Scalingregression testing to large software systems. InProceedings of the 12th ACM SIGSOFT Symposiumon the Foundations of Software Engineering (FSE2004) (Newport Beach, CA, 2004). To appear.
[18] Ramalingam, G., Field, J., and Tip, F. Aggregatestructure identification and its application to programanalysis. In Proc. of the ACM SIGPLAN-SIGACTSymp. on Principles of Programming Languages(January 1999), pp. 119–132.
[19] Ren, X., Shah, F., Tip, F., Ryder, B. G.,Chesley, O., and Dolby, J. Chianti: A prototypechange impact analysis tool for Java. Tech. Rep.DCS-TR-533, Rutgers University Department ofComputer Science, September 2003.
[20] Rothermel, G., and Harrold, M. J. A safe,efficient regression test selection technique. ACMTrans. on Software Engineering and Methodology 6, 2(April 1997), 173–210.
[21] Ryder, B. G., and Tip, F. Change impact for objectoriented programs. In Proc. of the ACMSIGPLAN/SIGSOFT Workshop on Program Analysisand Software Testing (PASTE01) (June 2001).
[22] Sarma, A., Noroozi, Z., and van der Hoek, A.Palantir: Raising awareness among configurationmanagement workspaces. In Proc. of the InternationalConf. on Software Engineering (2003), pp. 444–454.
[23] Steyaert, P., Lucas, C., Mens, K., and D’Hondt,T. Reuse contracts: Managing the evolution ofreusable assets. In Proc. of the Conf. onObject-Oriented Programming, Systems, Languagesand Applications (1996), pp. 268–285.
[24] Thione, G. L. Detecting semantic conflicts in parallelchanges, December 2002. Masters Thesis, Departmentof Electrical and Computer Engineering, University ofTexas, Austin.
[25] Thione, G. L., and Perry, D. E. Parallel changes:Detecting semantic interference. Tech. Rep.ESEL-2003-DSI-1, Experimental Software EngineeringLaboratory, University of Texas, Austin, September2003.
[26] Tip, F. A survey of program slicing techniques. J. ofProgramming Languages 3, 3 (1995), 121–189.
[27] Tonella, P. Using a concept lattice of decompositionslices for program understanding and impact analysis.IEEE Trans. on Software Engineering 29, 6 (2003),495–509.
[28] Zeller, A. Yesterday my program worked. Today, itdoes not. Why? In Proc. of the 7th European SoftwareEngineering Conf./7th ACM SIGSOFT Symp. on theFoundations of Software Engineering (ESEC/FSE’99)(Toulouse, France, 1999), pp. 253–267.
