Slides

1

Recommendation Systems for Code Reuse

Tao XieDepartment of Computer Science

North Carolina State University

Raleigh, USA

2 2

Motivation• Programmers commonly reuse

APIs of existing frameworks or libraries

– Advantages: Low cost and high efficiency of development

– Challenges: Complexity and lack of documentation

E.g., searching for information nearly ¼ of developer time [metallect.com]

Frameworks

Example Task from Eclipse ProgrammingTask: How to parse code in a dirty editor of Eclipse?

?Query:

“IEditorPart -> ICompilationUnit”

Open Source ProjectsOpen Source Projects

1 2 N…

… Extract

MIS 1MIS 2

...…

MIS k

*MIS: Method-Invocation sequence, FMIS: Frequent MIS

FMIS 1FMIS 2

…FMIS nRecommend

Mine

PARSEWeb [Thummalapenta&Xie ASE 07]

4

Scenario 1

• While reusing APIs of existing open source frameworks or libraries, programmers often – know what type of object they need – but do not know how to write code

for getting that object

Query: “Source Destination”

How to use these

APIs?

Prospector [Mandelin et al. PLDI 05 ], XSnippet [Sahavechaphan&Claypool OOPSLA 06 ], PARSEWeb [Thummalapenta&Xie ASE 07]

5

Example Task from Eclipse Programming

• Task: How to parse code in a dirty editor?• Query: IEditorPart ICompilationUnit

• Example solution from Prospector/PARSEWeb:IEditorPart iep = ...IEditorInput editorInp = iep.getEditorInput();IWorkingCopyManager wcm = JavaUI.getWorkingCopyManager();ICompilationUnit icu = wcm.getWorkingCopy(editorInp);

• Difficulties: a. Needs an instance of IWorkingCopyManager b. Needs to invoke a static method of JavaUI for getting the preceding instance

Prospector [Mandelin et al. PLDI 05 ], XSnippet [Sahavechaphan&Claypool OOPSLA 06 ], PARSEWeb [Thummalapenta&Xie ASE 07]

6

Scenario 2

• While reusing APIs of existing open source frameworks or libraries, programmers often – know what method call they need– but do not know how to write code

before and after this method call

Query: “Method name”

How to use these

APIs?

MAPO [Xie&Pei MSR 05]

7

Example Task from BCEL Programming

• Task: How to instrument the bytecode of a Java class by adding an extra method to the class?

• Query: org.apache.bcel.generic.ClassGen public void addMethod(Method m)

• Example solution from MAPO: public void generateStubMethod(ClassGen c) InstructionList il = new InstructionList(); MethodGen m= genFromISList(il); m.setMaxLocals(); m.setMaxStack(); c.addMethod(m.getMethod()); System.out.println(“…”); … }

MAPO [Xie&Pei MSR 05]

8

Scenario 3

• While reusing APIs of existing open source frameworks or libraries, programmers often – know structural context such as a

class’ type, its parents, and fields’ types, a method’s signature, method or constructor callees

– but do not know how to write code in this context

Query: Structural context

How to use these

APIs?

Strathcona [Holmes et al. 05], XSnippet [Sahavechaphan&Claypool OOPSLA 06 ]

9

Example Task from HttpClient Programming

• Task: How to evolve a system to use a third party library, HttpClient, for handling http connections?

• Query: HttpClient, PostMethod classes

• Example solution from Strathcona:

Strathcona [Holmes et al. 05], XSnippet [Sahavechaphan&Claypool OOPSLA 06 ]

10

Steps in Recommenders

• Data collection/extraction

• Data preprocessing

• Data analysis/mining

• Result postprocessing

• Result representation

11

Data Collection/Extraction

• From one or multiple local code repositories– Often followed by offline analysis or mining– Challenges: lack of relevant code examples– Ex.: Strathcona, Prospector, XSnippet

• From the whole open source world with a code search engine!– Often followed by on-the-fly analysis and mining– Challenges: only partial code files– Ex.: MAPO, PARSEWeb

12 12

Exploiting A Code Search Engine• Accepts queries including keywords of classes or/and

method names

• Interacts with a code search engine such as Google code search to gather related code samples

• Stores gathered code samples (source files) in a local code repository (later being analyzed and mined)

• Challenges: gathered code samples are partial and not compilable as code search engines retrieve individual source files instead of entire projects

PARSEWeb [Thummalapenta&Xie ASE 07]

13 13

Available Code Search Engines

• Google Code Search http://www.google.com/codesearch

• Krugle: http://www.krugle.com/• Koders: http://www.koders.com/• Codase: http://www.codase.com/• JExamples: http://www.jexamples.com/

etc.,

Why not using just code search engines?

What are Developers Searching for?

Assieme [Hoffmann et al. UIST 07]

339 sessions related to Java programming

15 million queries of Windows Live Search from May 2006.

117 API sessions (34.2%); 70 trouble-shooting sessions (20.6%)

15 15

API-related Search Sessions

• 64.1% sessions contained queries that were merely descriptive but did not contain actual names of APIs, packages, types, or members.

• The remaining sessions contained – API or package names (12.8%),– Type names (17.9%) – Method names (5.1%).

• Among all these API-related sessions, 17.9% contained terms like “example”, “using”, or “sample code”

Assieme [Hoffmann et al. UIST 07]

16 16

An Example 4-Query Session

• java JSP current date• java JSP currentdate• java SimpleDateFormat• using currentdate in jsp

Assieme [Hoffmann et al. UIST 07]

Only compatible with new Java versions

Why Not Use Web Search Engines?

Requires installation of external library,but no link

Code on pages essentially the same

Contains no code examples

parse xml java

©Raphael HoffmannAssieme [Hoffmann et al. UIST 07]

Code Search Engines

import javax.xml.parsers.*;import org.w3c.dom.*;public class JAXPSample {  public static void main(String[] args) {    String filename = "sample.xml";             try {      DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();      DocumentBuilder parser = factory.newDocumentBuilder();      Document d = parser.parse(filename);    } catch (Exception e) {      System.err.println("Exception: " + e.getMessage());    }  }}

Index source code of open-source Projects (from compressed archiveFiles and CVS repositories)

Code is parsed and terms in typenames, variable names, etc. areweighted differently.

©Raphael HoffmannAssieme [Hoffmann et al. UIST 07]

Why not use code search engines only?

Irrelevant(An Emacs Lisp File!?!)

Code is complicated, contains no comments related to query,

and is more than 300(!) lines long

Requires installation of external library,but no link

Code on pages essentially the same

parse xml java

©Raphael HoffmannAssieme [Hoffmann et al. UIST 07]

Why not use code search engines only?

MAPO [Xie&Pei MSR 06]

21

Steps in Recommenders

• Data collection/extraction

• Data preprocessing

• Data analysis/mining

• Result postprocessing

• Result representation

22

Fact Extraction

• Whole-program analysis: applicable when the whole code bases are available and compilable

• Partial-program analysis: applicable when only partial code samples are available and not compilable– When a code search engine is used

23 23

Analysis of Partial Code Samples• Not all code samples contain main method or

driver code that can serve as an entry point– consider all public methods as entry points

• Deal with local method calls by inlining methods

• Deal with conditionals/loops by traversing control flow graphs

• Deal with unknown types with heuristicsPARSEWeb [Thummalapenta&Xie ASE 07]

24

Type Heuristics I

• Inferring fully qualified class namesimport javax.jms.QueueSession;

import java.util.*;

Public class test {

public QueueSession qsObj;

public Integer intObj;

public Iterator iter;

…

- Fully qualified name of QueueSession is “javax.jms.QueueSession”, inferred through lookup of import statement

- Fully qualified name of Integer is “java.lang.Integer”, inferred through loading of a class by appending “java.lang” to the class name

- Cannot infer the fully qualified name of “Iterator” (incorporating domain knowledge of java.util helps)

24PARSEWeb [Thummalapenta&Xie ASE 07]

25

Type Heuristics II

• Infer the receiver type in expression “X.Y”– Lookup the declaration of X in local variables or

member variables. If not, “X” is a class name and Y is a static member

• Infer the receiver type in expression “M1().Y”– Check the return type of M1() method declaration, if

not available locally, the receiver type cannot be inferred

25PARSEWeb [Thummalapenta&Xie ASE 07]

26

Type Heuristics III

• Infer the return type of a method invocation in an assignment statement such as “Queue qObj = createQueueSession()”

– Lookup the type of the variable on the left hand side. The return type is the same as or a sub class of Queue

• Infer the return type of a method invocation in a return statement such as public QueueSession test(){ ...

return connect.createQueueSession(false,int);}

- Lookup the return type of the enclosing method declaration

26PARSEWeb [Thummalapenta&Xie ASE 07]

27

Type Heuristics IV

• Infer types with multiple method invocations

Queue qObj = connect.m1();

Stack sObj = connect.m1().m2();

The receiver type of m2() can be inferred from the lookup of the return type of m1()

27PARSEWeb [Thummalapenta&Xie ASE 07]

28

Sequence Filtering

• Remove common Java library calls• Remove sequences that contain no query

words: ClassGen and addMethod

InstructionList.<init>()

genFromISList(InstructionList)

MethodGen.setMaxStack()

MethodGen.setMaxLocals()

MethodGen.getMethod()

ClassGen.addMethod(Method)PrintStream.println(String) …

public void generateStubMethod(ClassGen c) InstructionList il = new InstructionList(); MethodGen m= genFromISList(il); m.setMaxLocals(); m.setMaxStack(); c.addMethod(m.getMethod()); System.out.println(“…”); …

}

MAPO [Xie&Pei MSR 05]

Type Signature Graph

Any path from h to w is a (h,w)-jungloid

IFile CompilationUnit

ICompilationUnit

ASTNode

IClassFile

JavaCore.createCompilationUnitFrom()

AST.parseCompilationUnit()supertyp

e

AST.parseCompilationUnit()

JavaCore.createClassFileFrom()

IJavaElement IResource

supertype

getResource()

IContainer

getParent()

Prospector [Mandelin et al. PLDI 05 ]

Jungloids with Downcasts

IDebugView debugger = ...Viewer viewer = debugger.getViewer();IStructuredSelection sel = (IStructuredSelection) viewer.getSelection();JavaInspectExpression expr = (JavaInspectExpression) sel.getFirstElement();

IDebugView

Viewer

ISelection

IStructuredSelection

JavaInspectExpressionObject

getViewer()

getSelection()

getFirstElement()

getIn

put()

downcast

downcast

Prospector [Mandelin et al. PLDI 05 ]

31

Steps in Recommenders

• Data collection/extraction

• Data preprocessing

• Data analysis/mining

• Result postprocessing

• Result representation

32

Data Analysis/Mining• Some recommenders don’t use specific

mining techniques to “abstract” or “generalize” common patterns but return relevant raw code samples– Prospector, Strathcona, XSnippet, PARSEWeb

• Data mining can be used to uncover hidden patterns– Association rules: CodeWeb [Michail ICSE 00]– Frequent subsequences: MAPO [Xie&Pei MSR 06]– Frequent partial orders: Apiator [Acharya et al. FSE 07]

33

Association RulesKApplication reuse patterns

CodeWeb [Michail ICSE 00]

#include <abcdef.h>void p ( ) { b ( ); c ( ); }void q ( ) { c ( ); b ( ); }void r ( ) { e ( ); f ( ); }void s ( ) { f ( ); e ( ); }

int main ( ) { int i, j, k; a ( ); if ( i == 1) { f ( ); e ( ); c ( ); exit ( ); } else { if ( j == 1 ) p ( ); else q ( ); d ( ); if ( k == 1 ) r ( ); else s ( ); } }

Frequent SubSeq/Partial Order

Consider APIs a, b, c, d, e, and f

Apiator [Acharya et al. FSE 07]

#include <abcdef.h>void p ( ) { b ( ); c ( ); }void q ( ) { c ( ); b ( ); }void r ( ) { e ( ); f ( ); }void s ( ) { f ( ); e ( ); }

int main ( ) { int i, j, k; a ( ); if ( i == 1) { f ( ); e ( ); c ( ); exit ( ); } else { if ( j == 1 ) p ( ); else q ( ); d ( ); if ( k == 1 ) r ( ); else s ( ); } }

1 a f e c2 a b c d e f3 a c b d e f4 a b c d f e5 a c b d f e

a

d

c

e

b

f

a b d e a b d fa c d ea c d f

(b) Static program traces (c) Frequent sequential patternsSupport 4/5

(d) Frequent partial order R(a) Example code

Consider APIs a, b, c, d, e, and f

Frequent SubSeq/Partial Order

Apiator [Acharya et al. FSE 07]

#include <abcdef.h>void p ( ) { b ( ); c ( ); }void q ( ) { c ( ); b ( ); }void r ( ) { e ( ); f ( ); }void s ( ) { f ( ); e ( ); }

int main ( ) { int i, j, k; a ( ); if ( i == 1) { f ( ); e ( ); c ( ); exit ( ); } else { if ( j == 1 ) p ( ); else q ( ); d ( ); if ( k == 1 ) r ( ); else s ( ); } }

1 a f e c2 a b c d e f3 a c b d e f4 a b c d f e5 a c b d f e

a

d

c

e

b

f

a b d e a b d fa c d ea c d f

(b) Static program traces (c) Frequent sequential patternssupport, 4/5

(d) Frequent partial order R(a) Example code

Frequent SubSeq/Partial Order

Consider APIs a, b, c, d, e, and f

Apiator [Acharya et al. FSE 07]

1 a f e c2 a b c d e f3 a c b d e f4 a b c d f e5 a c b d f e

a

d

c

e

b

f

a b d e a b d fa c d ea c d f

(b) Static program traces (c) Frequent sequential patternssupport, 4/5

(d) Frequent partial order R(a) Example code

#include <abcdef.h>void p ( ) { b ( ); c ( ); }void q ( ) { c ( ); b ( ); }void r ( ) { e ( ); f ( ); }void s ( ) { f ( ); e ( ); }

int main ( ) { int i, j, k; a ( ); if ( i == 1) { f ( ); e ( ); c ( ); exit ( ); } else { if ( j == 1 ) p ( ); else q ( ); d ( ); if ( k == 1 ) r ( ); else s ( ); } }

Frequent SubSeq/Partial Order

Apiator [Acharya et al. FSE 07] MAPO [Xie&Pei MSR 05]

MAPO

Apiator

38

Data Analysis/Mining

• Data collection/extraction

• Data preprocessing

• Data analysis/mining

• Result postprocessing

• Result representation

39

Result Postprocessing

• When a third-party miner or learner isn’t used, this step may be considered part of the data analysis/mining step.

Examples

• Result clustering

• Result ranking

• Result filtering

40 40

Clustering and Ranking

• Candidate method sequences produced by the data analysis/mining step for query “Source Destination” may be too many

Solutions:• Cluster similar sequences

– Clustering heuristics are developed

• Rank sequences– Ranking heuristics are developed

PARSEWeb [Thummalapenta&Xie ASE 07]

41 41

Clustering Heuristics

• Method-invocation sequences with the same set of statements can be considered similar, although the statements are in different order.e.g., ''2 3 4 5'' and ''2 4 3 5 ''

• Method-invocation sequences with minor differences measured by an attribute cluster precision value can be considered similar.e.g., ''8 9 6 7'' and ''8 6 10 7 '' can be considered similar under cluster precision value one

PARSEWeb [Thummalapenta&Xie ASE 07]

42 42

Ranking Heuristics

• Heuristic 1: Higher frequency -> Higher

rank

• Heuristic 2: Shorter length -> Higher rank

• Heuristic 3: Fewer package boundaries -> Higher rank

PARSEWeb [Thummalapenta&Xie ASE 07]Prospector [Mandelin et al. PLDI 05 ]

43

Query Splitting

• Lack of code samples that give candidate method-invocation sequences in the results of code search engines– Required method-invocation sequences are split among

different source files

• Solution:– Split the user query into multiple queries– Compose the results for each split query

PARSEWeb [Thummalapenta&Xie ASE 07]

44

Query Splitting Example1. User query: “org.eclipse.jface.viewers.IStructuredSelection->java.io.ObjectInputStream”

Results: None

2. Query: “java.io.ObjectInputStream” Results: 3.

Most used immediate sources are: java.io.InputStream, java.io.ByteArrayInputStream, java.io.FileInputStream

3. Three Queries to be fired: “org.eclipse.jface.viewers.IStructuredSelection-> java.io.InputStream” Results: 1

“org.eclipse.jface.viewers.IStructuredSelection-> java.io.ByteArrayInputStream” Results: 5

“org.eclipse.jface.viewers.IStructuredSelection-> java.io.FileInputStream” Results: None

PARSEWeb [Thummalapenta&Xie ASE 07]

45

Result Filtering• Remove sequences that contain no query

words: ClassGen and addMethod• Compress consecutive calls of the same

method into one, e.g., abbba aba• Remove duplicate frequent sequences

after the compression, e.g., aba, aba aba

• Reduce a seq if it is a subseq of another, e.g., aba, abab abab

MAPO [Xie&Pei MSR 06]

46

Data Analysis/Mining

• Data collection/extraction

• Data preprocessing

• Data analysis/mining

• Result postprocessing

• Result representation

47

Result Representation

• Display results in the tool user interface – Strathcona– XSnippet– PARSEWeb– MAPO– CodeBroker– Assieme

48 48

Strathcona

Strathcona [Holmes et al. 05]

49 49

XSnippet

XSnippet [Sahavechaphan&Claypool OOPSLA 06 ]

50 50

PARSEWeb

PARSEWeb [Thummalapenta&Xie ASE 07]

51 51

PARSEWebhttp://news.google.com/

52 52

MAPO (new)

MAPO [Xie&Pei MSR 06]

53 53

MAPO (new)

MAPO [Xie&Pei MSR 06]

CodeBroker

Comm

ents

signa

ture

CodeBroker [Ye&Fischer ICSE 01]

Information delivery that autonomously locates and presents software developers with task-relevant and personalized components. Active repository!!!

Assieme

• A hybrid search engine

• Index code snippets found on web pages

• Link them to required libraries and documentation

Assieme [Hoffmann et al. UIST 07]

Assieme

links to pages with snippets

group pages with similar snippets

links to required libraries

Assieme [Hoffmann et al. UIST 07]

Example Evaluations of Recommenders

• Prospector• Strathcona • PARSEWeb

Prospector Experiment 1 (ranking test)

• hypothesis: – to find the desired code, the user needs to

examine only top 5 candidate jungloids.

• result: – desired code in “top 5” 17 out 20 times (10

out of 20, in “top 1”)– remaining three fixable

• methodology:– used 20 real-world coding tasks– collected from FAQs, newsgroups, our

practice, emails to us

Prospector Experiment 2(user study)

• hypothesis:– Prospector-equipped programmers are better at

solving API programming problems than other programmers

• methodology: – 6 problems, each user did 3 with Prospector and 3

without– problems formulated not to reveal the query – sample problem:

“The new Java channel IO system represents files as channels. How do I get a channel that represents a String filename?”

– somewhat sparse data (10 users)

Experiment 2 (user study). Results.

• Prospector shortens development time– some problems solved only by Prospector users– when both groups succeeded, Prospector users

30% faster

• Prospector may help enable reuse– non-Prospector users sometimes reimplemented

• Prospector may help avoid making mistakes– mistakes applying code found on internet into

own code

• The authors expect even stronger results on a more robust infrastructure.

Strathcona: User Study• 2 developers were assigned 4 tasks on building a plug-in for

Eclipse. Neither developers knew how to implement any of the tasks at hand.

• The results showed that the tool can deliver relevant and useful examples to developers. They also showed a developer can determine when the examples returned are not relevant.

Table 2: Results from Evaluation:Useful Example Source Viewed Succeeded at Task

Task 1Subject 1 1 1 yesSubject 2 1 1 yesTask 2Subject 1 1 2 yesSubject 2 1 6 yesTask 3Subject 1 0 2 yesSubject 2 0 6 yesTask 4Subject 1 1 2 yesSubject 2 0 7 partially

Strathcona [Holmes et al. 05]

Strathcona: Performance and Scalability

• As a test case for scalability, Eclipse 3.0 source was populated to the repository. The resulting amount of information in the repository is shown in Table1.

• On a Pentium 3 800 MHz 1024 MB RAM Server, a Pentium 3 1000 MHz 256 MB RAM Repository with Postgresql DB the performance numbers are:

Table 1: Number of Structural Relations

Classes 17,456Methods 124,359Fields 48,441Inheritance Relations 15,187Object Instant ions 43,923Calls Relations 1,066,838

Total 1,316,204

– Less than 500 ms for building a structural context.

– Less than 300 ms for displaying the example.

– 4 – 12 seconds server response time.

Strathcona [Holmes et al. 05]

63

PARSEWeb Evaluations

• Real Programming Problems: To address problems posted in developer forums

• Real Projects: To show that solutions recommended by PARSEWeb are – available in real projects – better than solutions recommended by related tools PROSPECTOR,

Strathcona, and Google Code Search averagely

64

Real Programming Problems

Jakarta BCEL user forum, 2001

Problem: “How to disassemble java byte code”

Query: “Code Instruction”

Solution Sequence:FileName:2_RepMIStubGenerator.java MethodName: isWriteMethod Rank:1

NumberOfOccurrences:1

Code,getCode() ReturnType:#UNKNOWN#

CONSTRUCTOR,InstructionList(#UNKNOWN#) ReturnType:InstructionList

InstructionList,getInstructions() ReturnType:Instruction

Solution Sample Code: Code code;

InstructionList il = new InstructionList(code.getCode());

Instruction[] ins = il.getInstructions();

65

Real Programming Problems

Dev 2 Dev Newsgroups, 2006

Problem: “how to connect db by sessionBean”

Query: javax.naming.InitialContext java.sql.Connection

Solution Sequence: FileName:3 AddressBean.java MethodName:getNextUniqueKey Rank:1

NumberOfOccurrences:34javax.naming.InitialContext,lookup(java.lang.String)

ReturnType:javax.sql.DataSourcejavax.sql.DataSource,getConnection()

ReturnType:java.sql.Connection

66

Real Project: Logic• Source File: LogicEditor.java

SUMMARY-> PARSEWeb: 8/10, Prospector: 6/10, Strathcona: 5/10

67

Comparison with Prospector• 12 specific programming tasks taken from XSnippet approach.

SUMMARY-> PARSEWeb: 11/12, Prospector: 7/12

68

Comparison with Other Tools

Percentage of tasks successfully completed by PARSEWeb,

Prospector, and XSnippet

69

Significance of Internal Techniques

*Legend:Method inline: Method inliningPost Process: Sequence Post ProcessorQuery Split: Query Splitter

70T. Xie Mining Program Source Code

Questions?

Bibliography on Mining Software Engineering Data http://ase.csc.ncsu.edu/dmse/•What software engineering tasks can be helped by data mining?•What kinds of software engineering data can be mined?•How are data mining techniques used in software engineering?•Resources

Available Data Mining Toolshttp://ase.csc.ncsu.edu/dmse/resources.html

Mining Partial Orders

71

Consider APIs a, b, c, d, e, and f

Partial OrderPartial Order with

Transitive Reduction

The extracted scenarios are fed to a partial order miner

The partial order miner mines frequent closed partial order

Closed Partial Order

Apiator [Acharya et al. FSE 07]

XOpenDisplay

XCloseDisplay

XCreateWindow

XGetWindowAttributes

XCreateGC

XSetForeground

XGetBackground

XMapWindow

XChageWindowAttributes

XMapWindow

XSelectInput

XGetAtomName

XFreeGC

XNextEvent

Example Partial Order

A usage scenario around XOpenDisplay API as apartial order.

Specifications are shown with dotted lines.

Apiator [Acharya et al. FSE 07]