Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the fullcitation on the first page. Copyrights for components of this work owned by othersthan ACM must be honored. Abstracting with credit is permitted. To copy otherwise,or republish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee. Request permissions from Permissions@acm.org. PLATEAU'14, October 21 2014, Portland, OR, USA Copyright 2014 ACM 978-1-4503-2277-5/14/10…$15.00 http://dx.doi.org/10.1145/2688204.2688208

EUKLAS Supporting Copy-and-Paste Strategies for

Integrating Example Code

Christian Dörner

Senacor Technologies AG Erika-Mann-Str. 55

80636 München

christian.doerner@senacor.com

Andrew R. Faulring

HCI Institute Carnegie Mellon University

5000 Forbes Avenue, Pittsburgh, PA

faulring@cs.cmu.edu

Brad A. Myers

HCI Institute Carnegie Mellon University

5000 Forbes Avenue, Pittsburgh, PA

bam@cs.cmu.edu

Abstract Researchers have paid increasing attention in recent years to the fact that much development occurs though example modification. Helping programmers with some of the pit-falls and vagaries of working with example code is the goal of our tool, called Euklas. It helps developers to integrate JavaScript example code into their own projects by using familiar IDE interaction techniques of the Eclipse IDE. The Euklas plugin uses static, heuristic source code checks to highlight potential errors and to recommend potential fixes, when incomplete sections of code are copied from a work-ing JavaScript example and pasted into the program being edited. The most unique feature of the tool is the ability to automatically import missing variable and function defini-tions from an example file into a new project file. Our preliminary user study of Euklas suggests that it supports users in fixing errors more easily.

Categories and Subject Descriptors D.2.6 [Program-ming Environments]: Programmer workbench.

General Terms Algorithms, Design, Human Factors

Keywords Code Reuse, Copy-and-Paste, JavaScript, Eclipse, Natural Programming, Examples

1. Introduction Leveraging examples is an established technique in design [1], and has recently received increasing attention from

researchers focusing on programming tools [2-5]. With the rise of search engines and web repositories of code along with discussion threads, blogs, and code example websites, people often create new systems by copying and pasting code snippets from such sources [6]. Surveys and research show that looking for examples is often people’s preferred way to learn how to perform a task or to learn how to use application programming interfaces (APIs) [7].

Most of the research on reusing examples has focused on building improved search and data mining tools to help with finding the examples (e.g., [1, 3, 5, 7, 8]). However, there has not been much research on assisting users in reus-ing and integrating the examples after they have been

Figure 1. Euklas supports peoples’ integration attempts as it enhances Eclipse’s JavaScript editor by (#1) highlighting errors in the source code, (#2) providing quick fixes including the use of the context in the origin of any copied code, and (#3) including explanations for copy-and-paste errors based on the code of the example snippet that was used.

13

fb([ag

Uwpsapur

gvinsoswswswe

thshtyucirtinatapthavoe1fFv

1 n

found, with thebeen identified (e.g., copying s11-13]. Our a

attempts by prguidance to hel

This paper inUsers’ Keystrowhich helps uspaste strategiesspecific guidanand-paste errorpart of the requusers with reuresulting errors

JavaScript iguages for webvariety of usernteraction des

spite of this wioften provide lstatic code chewith other, nonson may be thaweak, dynamicstatic analyses well as provideerrors resulting

The followihe kind of cop

supported by Ehas found a coypes of enhan

use one in her code in more drrelevant pieceions, variables

necessary to mample. This isask especially

provide any hehe integration

ance. First, Euvariables, missiof CSS and Javerror markers 1). Second, Eukfied errors to Figure 1). For variable and fu

Euklas is Germa

nounced oy-class.

e exception of as a common

several lines, baim is to suproviding famililp them make tntroduces our

okes Lessened sers to more sus for reuse (sence for assistinrs that are typicuired code pat

using working s. is one of the b programmingrs, from end-uigners [6], to idespread use, less programmecking, limitedn-scripting langat analyzing Javc typing that mat edit-time. W

e more sophistg from copy-aning example ppy-and-paste reEuklas. Jamie ode snippet onnced combo-bo

website. Jamietail to separates. That is, sh, and imports o

make her code ws often a timey since Jamie’elp. In contrastprocess as it p

uklas highlighing function devaScript files,

and squigglyklas also comphelp Jamie eaexample, it s

unction definit

an for Euclase, w

[4, 9, 10]. Copusage pattern locks, or even

pport people’s iar, easy-to-ustheir copied cotool, called EUby Attaching uccessfully emee Figure 1). Eng users in fixcally caused byttern [11]. Thecode from ot

most populag [15] and is uuser developerprofessional pJavaScript de

ming support fod auto-completiguages such asvaScript is diff

makes it challenWe overcome tticated guidancd-paste. resents a typiceuse of JavaScis a JavaScrip

n the web that oxes from whiie has to explote relevant piec

he has to identof JavaScript awork the same-consuming an’s JavaScript t, Euklas can h

provides her whts errors, sucefinitions, and in her target coy underlines (s

putes quick fixeasily correct thsuggests copyitions from the

which is a gemsto

py-and-paste hafor reusing codwhole methodcopy-and-pas

e, and effectivde work.

UKLAS (Eclipfrom Samples)

mploy copy-andEuklas provide

xing some copyy copying onlye goal is helpinthers and to f

ar scripting lanused by a broars [16] such a

programmers. Ivelopment too

or users (e.g., nions), compares Java. One reficult since it hanging to perforthis limitation ace for correctin

cal use case focript code that pt developer an

creates variouch she wants ore the exampces of code frotify which funand CSS files ae way as the exnd cumbersomeditor does nhelp Jamie wiith editing guid

ch as undefinemissing imporode by insertinsee #1 in Figues for the identhem (see #2 ing the missinsource file an

one. Euklas is pr

as de

ds) ste ve

se )1, d-es y-

y a ng fix

n-ad as In

ols no ed a-as

rm as ng

for is

nd us to

ple m

nc-are x-

me not th d-ed rts ng re ti-in ng nd

ro-

offers and Ja

Eukusers tprojectpastedthe conwere cfeaturetion anheuristas chetions, awas coerrors.

Wepeopleresultsusableconfusful. Thwere ain the their ta

2. RThe relocatintarget findingthe weare toosystem

JDAusers tfrom Jwrite pieces UNIX Script not accEuklasusers ttheir c

D.MHTMLbles thexistinit follorepresements’D.MIXone st

to insert the mavaScript files.klas makes theto integrate Jats by identifyi

d code and recnsideration of

copied. The edes of the Eclipnd repair algortic edit-time so

ecking for uninand the analysiopied) to provi. e evaluated Euke, comparing s suggest that the and effectivesed that Euklashe study also sable to fix abo

control grouparget systems.

Related Weuse of examplng the examplsystem. Thereg relevant exaeb) [2, 3, 17, 18ols that suppo

ms. A (JavaScript that have no prJavaScript codesimple HTML of JavaScript systems. Howcode as “blac

cess or changes treats all Javato access and c

code structure.MIX targets wL and scriptinghem to build ng web sites. Dows a copy-andentation of the’ parameters cX’s editing envtep further, si

missing import

e following maavaScript examing some of thcommending pf the original cditing guidancepse IDE and exrithms. These ource code che

nitialized variabis of the exampide useful quic

klas in a prelimit to Eclipse’she features have manner, sincs can sometimsuggests that p

out two times ap, when integr

Work le code consistle code and 2

e are several syample code (e.8]. However, mort users in int

Dataflow Archrogramming ske that was founL commands t code in a wwever, JDA trk-boxes”, whi

e the internal coaScript code ascopy even inco

web designersg languages, suand share ma

D.MIX inspiredd-paste approae selected datacan be changevironment. Weince Euklas p

t statements fo

ajor contributiomple code intohe potential erpotential fixesode from whic

e uses familiar xtends it with nalgorithms are

ecks for JavaSbles and undefple file (from wck fixes for the

minary user stus JavaScript eve been implemce participants

mes be wrong oparticipants usas many errorsrating example

ts of two main2) integrating ystems that ass.g., in reposito

more relevant ttegrating code

hitecture) aimskills to build apnd on the web to connect th

way similar to reats the piecech means that ode structure. Is “glass boxes”omplete interna

s that are famuch as JavaScrashups created d the design ofach that works a [19]. In thated after pastine take the idea provides speci

or the CSS

on: It helps o their own rrors in the , based on ch sections interaction new detec-e based on

Script, such fined func-where code e identified

udy with 12 editor. The mented in a s were not or not help-ing Euklas s as people e code into

n phases: 1) it into the ist users in ories or on to our work e into their

s to enable pplications [10]. Users e different “pipes” in

es of Java-users can-

In contrast, ”, allowing al pieces of

miliar with ript. It ena-

from pre-f Euklas as on a richer t way, ele-ng them to

of D.MIX fic editing

14

oin

rthdoainupw

c[dineoetasdc

cgmdI

mlsOgaafSinaqck

Rurpfineth

operations, in tntegrate their c

The Lookingreuse functionahem to unders

dents to do thisof the programand end of thentegrate this f

uses the idea opaste-based reuwith familiar in

JIGSAW is copy-and-paste9]. It inspired

developers withnto the develop

example code aof the example ever, Jigsaw carget code tha

structures. Jigsdevelopers, whcumbersome an

JSLINT is acript code. JSLgoal is to helpmakes it differdo not restrict In addition, Euk

Many JavaSming support oevel, IDEs like

such as checkinOne of the mogramming Javaadvanced codeample, it offerfunctions haveStorm also offeng users to eas

a new empty quick fixes do copied, makingkind of quick fi

Euklas’s copRosson’s and uses in Smalltreuse strategy piece of code tfunctionality thnto the target

editing directiohe code work

the form of qucopied pieces og Glass IDE hality they find stand how the s by enabling th

m they are intee functionality functionality inof guiding useuse strategies,nteraction techn

a plug-in fore interaction ted the design ofh the integratiper’s own sourand the target code would fian only work

at have similarsaw was testedhich showed thand not yet wella popular tool Lint works on p programmerrent from Euklwhich featuresklas supports u

Script editors dother than syne NetBeans prong whether a vost advanced caScript is Webe inspections brs analyses to been defineders quick fixes sily declare andfunction definnot analyze th

g it impossiblixes that Euklapy-and-paste dCarroll’s obsetalk programmwhere progra

that they consihat they intend

context, theyons about whatk. This was o

uick-fixes, whiof code into theelps middle scin other progrcode works [4hem to 1) reco

erested in, 2) ithey are intern their new p

ers to support but integrateniques in Eclipr the Eclipse Ichnique for ref Euklas, sincion of the reusrce code. Jigsacode to sugge

it best in the tak with pieces or AST (Abstrad in a small at the resolutio supported by for detecting a subset of Jars to write belas’s error detes of the languausers in fixing tdo not provide ntax highlightiovide simple cvariable has becommercial prbStorm, whichby leveraging

check whethed. In contrast t

for the detected define a varianition. Howevehe files from we for WebSto

as provides. design is stronervations abouming” [13]. Thammers copiedidered to be prded to reuse.

y let the envirt would be nec

often accompli

ch help users eir projects. chool students rams by helpin4]. It guides sturd the executiodentify the sta

rested in, and 3program. Eukla

their copy-andes this guidancpse. IDE that uses

eusing Java code Jigsaw assissed source codaw compares thest which piecrget code. Howof example an

act Syntax Trestudy with tw

on of conflicts Jigsaw. errors in JavaS

avaScript and ietter code. Thection, since wage can be usethe errors. much programng. At the neode inspectioneen used or noroducts for proh provides moJSLint. For ex

er variables anto JSLint, Webed errors, allowable, or to creaer, the providewhere code wa

orm to offer th

ngly inspired but “the reuse ohey observed d and pasted romising for thAfter pasting onment providcessary to makished in sever

to

to ng u-on art 3) as d-ce

a de sts de he es w-nd e)

wo is

S-its his we ed.

m-xt

ns, ot. o-re x-nd b-w-ate ed as he

by of a a

he it

de ke ral

cycles until a

Kimenced ly coptionalievidenoften spend

Eukcopyinwill nofor sompastingare pehave b

3. EIn thissign. Aplemen

3.1 U

Returnexplaincode s

structocode fpastes two ercalls fThese not avcopiediar maerror min the

FigurJamie

of fixing andall of the copiedm et al. reportfields/constant

pied together bity [12]. In adnce that the idfails in the firtime on identif

klas supports ung-and-pastingot only mark eme of these errg additional pi

erformed iteratibeen fixed.

EUKLASs section we fiAfterwards wented and integr

User Interface

ning to the exan how Euklas hshe found on t

or (jawBar(idfor making ththe constructo

rrors in the cofindMatch(eare undefined

vailable in the td from the exaarker system inmarkers in tcode.

re 2. Euklas me pastes it into th

d waiting for nd code was fixet that related cts and caller/ca

because they bddition, Ko etdentification orst attempt, refying all relevausers when th

g it into their terrors, but it arors (which miieces of the exaively until all

first discuss Eue will present hrated into the E

e

ample discussehelps Jamie tothe web. Jamie

d), see Figure 2he combo-boxor into her targonstructor: thee) and init(d because theirtarget file, sinc

ample file. Eukncluding the squthe margin to

marks errors in he target file.

new editing sued. code snippets allee methods)

belong to the st al. providedof related codequiring progrant pieces of cohey try to reustarget code sinalso provides qight involve coample code). Terrors in the t

uklas’s user inhow the systemEclipse IDE.

ed in the introd integrate the ce identifies tha

2) is a promisinx work. She cget file. Euklas two undefine) (see #1 inr function defice they have noklas uses Eclipuiggle underlinindicate these

the function ja

uggestions,

(e.g. refer-) are usual-same func-d empirical de snippets rammers to ode [11]. se code by nce Euklas quick fixes opying and These steps target code

nterface de-m was im-

duction, we combo-box at the con-

ng piece of copies and s identifies ed function

Figure 2). initions are ot yet been

pse’s famil-nes and the two errors

awBar() after

15

s

eff3f

fthfthfbwbbfwato

FeoltTtisthu

3

EeH

Ffpsffw

In addition solutions for fi

that refers toerror, as shownfunctionality fofixes proposed 3) use Euklas’sfixes proposed

The first qufunction jawBahe example fi

function jawBahat Euklas pro

function, calledbody. This is swould not servbo-box work inbecause it resefered by Eclipwhich programalso a fallback o provide a bet

Jamie readsFigure 3) that error by using option is probaects it. Eukltype.init() This successfulion init().

still undefined he newly code

until the code is

3.2 Impleme

Euklas was imeditor as a pluHTML editors

Figure 3. Euklafined function inproposed fix is shshows a short pfollowed by a prefix has been appwhich is well-kn

to marking thixing them. Byo init(), Jamn in Figure 3. or showing theby Euklas (the

s icon to diby Eclipse itseick fix that Euar.prototyp

file (the file frar()) to the taoposes is a “dd init() that sufficient to ree Jamie’s goal n her website. embles the kinpse’s Java ed

mmers may be option for castter quick fix b

s the explanatidescribes howthe example c

ably the correclas pastes the

from the exally fixes the mHowever, the and Euklas id

e. Jamie contins fixed and the

ntation

mplemented onug-in to the IDdo not provid

as proposes the tnit(). More inforhown in the beigproposal explaineview of how thlied. Euklas augown by many pr

e errors, Euklay clicking on t

mie gets four opAgain, Eukla

e quick fix opte first two in tstinguish themelf (the last twouklas proposes pe.init() thfrom which Jaarget file. The sdefault fix” thawould have anemove the synof making theWe added thi

nds of quick fiitor and by Wfamiliar. The des in which Eu

based on the exion in the beigw Euklas propcode. She decidct fix for her se function j

ample file intomissing definiti

function findentified an adues to fix the r

e combo-box is

n top of EclipDE. Eclipse’s

de as much pro

top two quick firmation about thge pop-up at the ning what Euklhe code will lookgments Eclipse’srogrammers.

as also suggesthe error markptions to fix ths uses Eclipsetions. The quicthe list in Figu

m from the quico in the list). would copy that is defined

aime copied thsecond quick fat creates a nen empty functiontax error, but e enhanced comis default optioixes that are oWebStorm widefault option uklas is not ab

xample code. ge window (seposes to fix thdes that the firsituation and sawBar.proto

o the target filion of the funndMatch(e) dditional error remaining erros working.

pse’s JavaScriJavaScript an

ogramming sup

fixes for the undhe first (selecteright. The pop-uas intends to d

k after the selectes quick fix featu

sts ker his e’s ck re ck

he in he fix ew on it

m-on of-th is

ble

ee he rst e-o-le.

nc-is in

ors

pt nd p-

port. Isyntaxin file”

Javto reliavariounon-fuan und‘null’ for adoperatithat artionshiin the can bein a fuvariablmissindefinitwhich HTMLfunctio

EukSyntaxcode. pastes file. Etypes providbe the runtimerrors cause do not

Eukused in[21]. SvariablEuklasen to [22]. Tof thecorrecters a utype). errors undefidetectinumbeerrors

Theare monumbedynam

de-ed)up

do,ed

ure

In particular, thx errors and so” and “assign s

vaScript is a dyably detect err

us situations thaunction value (defined variabor ‘undefined’

dditional errorstions. We idenre specific to Jaip with HTMLfollowing list

e detected by Eunction’s paramle definitions,

ng CSS style tions), 5) missimight be loca

L elements beon getElemenklas employs hx Tree (AST) Euklas’s analya piece of co

Euklas providementioned ab

des quick fixes more importan

me errors in thein the HTMLproblems with

t necessarily leklas’s code anan other static Since JavaScrles, it is imposs instead emplobe a successf

The implemente used exampt, which mean

useful result (aEuklas uses thlisted above. T

ined variables ion algorithm er 3), and the tin the HTML c

e heuristics useore fragile thanered 1-3. The

mically, e.g. loa

hey only proviome basic quicstatement to neynamic languarors at edit timat can cause ru(e.g. undefinedble, and acces’ [20]. Startings that may arintified three adavaScript, and L and CSS codof potential co

Euklas: 1) missmeter list, 2) m, 3) missing sheet imports ing JavaScript ated on remoteing accessed bntById(HTML_heuristic, static

to find potenysis of the A

ode from an exes error highlibove. However

for numbers 1nt types of erroe JavaScript c

L part of the sh displaying thad to JavaScripalyses are diffcode analysis

ript does not ssible to run coys heuristic aful alternative tation of Eukla

ples is syntactns that the codea limitation of three different aThe first detect

(error numbechecks for un

third detection code (error numed for identify

n the heuristicsreason is tha

ading CSS cod

ide error highlk fixes, such a

ew local variabage, which mame. Jensen et auntime errors: d) as a functiossing a prope

g with this list, ise from copy dditional types

d arise from its de. Our analysopy and paste ing parameter missing local function defin(e.g. for genefile imports (e

e servers), and by the global _Element_ID analyses on th

ntial errors in ST is invokedxample file toighting for allr, Euklas curr1-3, which we cors, since they code. Numberssource code, whe webpage copt execution-timferent from tho

tools, such asprovide static

certain dataflowanalyses, which

approach for as assumes thatically and see is executablethe current Eualgorithms to ition algorithm

ers 1 and 2), tndefined functalgorithm che

mbers 4-6). ying errors nums used for detecat content can de within JavaS

lighting for as “rename

ble.” kes it hard al. identify invoking a

on, reading erty that is

we looked y and paste s of errors close rela-

sis resulted errors that definitions and global nitions, 4) eral layout e.g. scripts, 6) missing JavaScript

D). he Abstract JavaScript

d if a user o the target l six error rently only consider to may cause

s 4-6 cause which may orrectly but me errors.

ose that are s FindBugs

typing of w analyses. h has prov-JavaScript

at the code emantically and deliv-

uklas proto-identify the checks for

the second tions (error ecks for the

mbered 4-6 cting errors

be loaded Script code.

16

Trm

pmeCecttoctfto

athscrrEmeths

estiusus

seEvfsqfmfc

4TCtu

To visualize ouristics, error numarker inste

There are spositives or falmeans that the error, or miss Consider the ferrors in the evcaused by “hthat.findMato detect that

could be relatethat. It wouldfind the functioo the object th

Euklas not oalso can propohe links betwe

specific region code. Each targregion has exaregion in an exEuklas maintaimemory. It upedited to keep this metadata a

shuts down. Currently, E

example files, ispace. We deciion for the Euk

usability and ushould handle eual snippets ofsmall pieces of

The ASTs osolutions for serror in the taEuklas analyzevariable (underfile(s) is syntacsuitable definitquick fix propofile into the tarmarker in the tafix, Euklas parcopied AST pie

4. EvaluaThe evaluationCan users undures? Is the in

ur confidence umbers 4, 5, anad of an error mome cases in se negatives, danalyses can emarking some

following examvaluation. Euklahiding” variabtch(e);. In t

the function d to the object

d produce a falon definition ofhat . only finds potese fixes for theen the positioin the target f

get file can haactly one link xample file fromins the meta-dapdates the regithe metadata c

as part of the p

Euklas needs ti.e., they mustided that this wklas prototype,usefulness of example code f code (e.g., cf code in blog pf the example

some classes oarget file refees the exampler the assumptioctically and semtion in an ex

osal to copy tharget file, and aarget file. If therses the AST oece from the ex

ation n had the goalderstand and sntegration of e

level of the md 6 are markedmarker . which Euklas

due to the usedeither mark coe existing erromple that cauas cannot detec

bles: var ththis case Eukla

call that.ft this insteadse positive becf findMatch(

ential errors inhese errors. Euon in the examfile when usersave multiple reto exactly onem which the cata about theseions when the correct. Euklas project when E

to have accessbe present in

would be a re, to enable invits ideas, butfrom external

code copied frposts). files are analyz

of errors. For ers to an und file(s) for a d

on that the codemantically corrxample file, Eat definition fr

adds this quicke user selects aof the target fxample file.

l of answeringuccessfully us

example code

more fragile heud with a warnin

s produces fald heuristics. Thorrect code as aors in the codused one of thct errors that ahat = thisas is not be abindMatch(e) of to the objecause it does n(e) that belong

n the code, but uklas remembemple file and ths copy and pasegions and eace correspondinode was copiee connections

target files aloads and savclipse starts an

s to each of thEclipse’s workasonable limitestigation of tht a real systefiles or individom the web, o

zed for potentiexample, if thefined variabl

definition of the in the exampect). If there is

Euklas creates rom the exampk fix to the erroa proposed quicfile to insert th

g two questionse Euklas’s fefaster and mo

u-ng

se his an

de. he

are s; ble ;

ect not gs

it ers he ste ch ng ed. in

are es nd

he k-ta-he m d-or

ial he le, his ple s a

a ple or ck he

ns: a-re

correcteditor tor, wh

4.1 P

We fehave snot waTherefyear olanguaprojectWe relocal usated $to 37 verse softwa

4.2 A

The stgroundstallatiThe codoes nexperim“Euklatween the tarwhile chose able towanteding andJava ehavingtook th

Weused foin Firetest wh

4.3 P

The stversionEclipslite” aner the subjecimposswould run. Psince a

t with Euklas tor compared t

hich already of

Participants

eel that Euklassome experienant to have tofore, we recruof experience age), and whot in JavaScriptcruited 12 paruniversity com$15 for participyears (medianbackgrounds,

are engineering

Apparatus and

tudy was conds. We used aion including tontrol group unot highlight emental group as lite” and “Ethe two versio

rget file to iden“Euklas full” to have these

o separately stud to explore thd quick fixes w

editor. Secondg the more sophe example filee set up Eclipsfor the study. Eefox to allow phether their tar

Procedure

tudy used a bn as indepene’s JavaScriptnd “Euklas ful tasks were c

cts design wassible for the pa

d have known Participants weall subjects rep

than with Eclipto a more sophffers some erro

s is most appnce with usingo train people uited participausing Eclipse

o had done at t (using any devrticipants (10 m

mmunity. Each pating. Their an: 25, s.d. 5).

such as busig.

d Materials

nducted in ouan iMac runninthe WTP (Web

used Eclipse’s errors or provid

used one of Euklas full”. T

ons was that “Entify errors andalso analyzed two different

udy the followihe effects of pwith a user inte, we were int

phisticated anale(s) into accouse’s workspaceEach of the exaparticipants to erget code was w

between-subjecndent variable t editor (Conll”. The dependcompleted sucs chosen becaarticipants to rthe answer to

ere randomly ported approx

pse’s standard histicated Javaor detection fea

propriate for pg JavaScript, a

on how to usants who had (for developileast one pro

velopment envmale, 2 femaleparticipant wages ranged froThe participan

iness administ

ur lab on the ng a standard Eb Tools ProjecJavaScript edide quick fixestwo versions The main diff

Euklas lite” onld to compute q

the example versions of Euing two aspect

providing errorerface similar toterested in thelyses and quick

unt. e with the filesamples could bexplore the exaworking. .

cts design usinwith three c

ntrol conditiondent variable w

ccessfully. Theause it would redo the tasks, each task aftassigned to eaimately equal

JavaScript aScript edi-atures?

eople who and we did se Eclipse. about one

ing in any ogramming vironment). e) from our as compen-m 19 years nts had di-tration and

university Eclipse in-ct) plug-in. itor, which s, and each of Euklas:

ference be-ly analyzed quick fixes, file(s). We uklas to be s. First, we r highlight-o Eclipse’s effects of k fixes that

s that were be executed amples and

ng the tool conditions:

n), “Euklas was wheth-e between-have been since they

er the first ach group, JavaScript

17

programming experience. Participants in all groups re-ceived a spoken introduction to the study and signed the consent form. All participants were briefly introduced to Eclipse’s JavaScript editor. Members of the experimental groups received an additional introduction to Euklas’s extensions to Eclipse’s JavaScript editor, e.g. information about its error and warning markers and the quick fixes. All task descriptions explained which code should be copied and what the desired results would be after the code was pasted and all errors were fixed.

All participants performed the tasks in the same order. The tasks were designed to cover all cases in which Euklas provides support, as well as cases where it provides mis-leading help or does not help at all. Participants were al-lowed to work on each task for a fixed amount of time, which varied from 7 to 20 minutes, based on the task’s difficulty. The researcher who conducted the study meas-ured the time it took participants to work on each of the tasks and stopped them if they ran over the maximum time allowed for each task. Task No. of

Errors Types of Errors*

Source LOC

Copied LOC

Max. Time (min.)

1 1 2 112 10 7 2 3 2, 3 112 13 7 3 2 1, 2 122 29 7 4 3 2, 5, 6 103 5 10 5 5 3, syntax

error 218 125 12

6 10 2, 3, 4 1507 996 20 Sum 24 2174 1178 63

Table 1. Summary of the tasks (*types of errors ac-cording to the list in section 3.2

Table 1 shows the number of errors per task (errors that occurred after the first paste operation), the types of errors included in the tasks (based on the types of errors presented in section 3.2), the lines of code (LOC) of the source ex-ample files (HTML and JavaScript files together), the total number of lines that had to be copied to solve the tasks, and the maximum time participants were given to complete each of the tasks. We judged tasks to be finished success-fully if the code could be executed without causing any errors in Firefox and if it performed as required by the specification.

In task #1, participants had to choose and integrate the correct part of a larger function for setting a cookie, while they had to choose and integrate a different part of that function for getting a cookie in task #2. For task #3, partic-ipants had to integrate an enhanced pop-up menu into the target file. In task #4, participants had to add a pushpin to a ‘Bing’ map and integrate it into the target file. To complete task #5, participants had to integrate an enhanced combo box (the one that we described above as Jamie’s example) into their target file. Finally, in task #6, participants had to integrate an inner window into their target file.

At the end of the study, participants filled out a ques-tionnaire. The questions primarily used five-level Likert scales, but some were open answer.

4.4 Results

The analysis of the data shows that participants using “Euklas full” completed more tasks (Control: 7/24, “Euklas lite”: 13/24, “Euklas full”: 18/24.) and fixed about twice as many errors (average = 22.25) as the control group did (average = 11.75) when integrating the example code into the target system.

For the analysis we combined the tasks shown in Table 1 into an “easy” group (tasks 1-3) and a “difficult” group (tasks 4-6). There are several reasons for combining the tasks into the two groups. The median average success rate for all six tasks across all three conditions was 0.5. We defined “easy” tasks as those with an average success rate at or above the median, and “difficult” tasks as those below the median. Another reason was that the tasks had been designed with an increasing difficulty. We allotted more time for completing the more difficult tasks. Finally, with-out combining the tasks we would have not been able to perform any statistical analysis, due to the highly differen-tiated completion values. The same reasoning justifies the statistical tests on the number of errors fixed. “Easy” Tasks “Difficult” Tasks 1: 0.75 4: 0.33 2: 0.50 5: 0.17 3: 1.00 6: 0.42

Table 2. Average success rate per task (std. error: 0.13) We ran a logistic regression analysis of the tasks, which

showed a statistically significant difference for the success rate with respect to the tasks’ difficulty, i.e. between the “easy” and “difficult” tasks: likelihood ratio χ²[1] = 14.79, p < 0.0001. We also analyzed how the success rate of par-ticipants was affected by the tool used (Control, Euklas lite, Euklas full) and the task difficulty (easy tasks, difficult tasks). A nominal logistic regression analysis showed that the effect of the tool on the success rate was significant: likelihood ratio χ²[2] = 14.97, p = 0.0006. The effect of the task difficulty on the success rate was also significant: likelihood ratio χ²[1] = 19.34, p < 0.0001. The interaction was not significant. In other words, participants were more successful based on the tool used across all tasks. We per-formed pairwise nominal logistic regression tests to deter-mine which levels of tool use had a significant effect on the success rate: Control vs. “Euklas lite”: likelihood ratio χ²[1] = 3.21, p = 0.074; Control vs. “Euklas full”: likeli-hood ratio χ²[1] = 14.90, p < 0.0001; “Euklas lite” vs. “Euklas full”: likelihood ratio χ²[1] = 4.44, p < 0.035. In summary, “Euklas full” was better than both Control and “Euklas lite”. “Euklas lite” was slightly better than Control.

Looking at the number of corrected errors, i.e. the num-ber of errors that were fixed by the participants, the data

18

also shows that “Euklas full” participants fixed almost twice as many errors as participants in the control group We ran an ANOVA to test for the effect of the tool used and the task difficulty on the number of corrected errors, and found that the number of corrected errors differed sig-nificantly across the three tools, F[2,9] = 13.82, p < 0.002. We ran contrasts to compare the tools with each other. The Control group made fewer error corrections than the “Euklas lite” group (F[1,9] = 13.37, p = 0.005) and also corrected less errors than the “Euklas full” group (F[1,9] = 26, p < 0.001).

Unfortunately, we were not able to analyze differences for the timing data, i.e. how long participants took to com-plete each of the tasks, since there were so many tasks that participants failed to complete in the maximum allotted time. The analysis of the final questionnaires provides more details on the differences between the two Euklas versions. Participants who used “Euklas full” agreed that it usually provided helpful quick fixes (average 4 out of 5). One “Euklas full” user nicely expressed why he liked it: “Intel-ligent error messages and debugging makes it infinitely more useful, especially when it checks against the source of your copy.” “Euklas lite” got lower ratings in terms of helpfulness (average 3.25 out of 5) of its quick fixes from the participants who used it, which was not surprising. One of the “Euklas lite” participants suggested the following improvement, which reflects exactly the improvements in “Euklas full”: “Provide [a] pop-up menu which can suggest to copy blocks of code to resolve errors.” The questionnaire also asked whether Euklas speeds up the integration of JavaScript code compared to other editors. Participants using “Euklas full” strongly agreed with this statement (4.75 out of 5) while participants using “Euklas lite” did not share this view (3.75 out of 5).

5. Discussion The tasks that were combined in the “easy tasks” group contained a maximum of three errors and participants were allotted the same maximum time of seven minutes for each of the tasks. Even though the number of errors was low and the scripts were short, participants in the Euklas conditions were more likely to complete the three tasks and fixed more errors than participants in the control group. However, due to the rather low complexity of tasks in this group, we did not expect that “Euklas full” participants would have big advantages compared to “Euklas lite” participants, which was reflected by the very similar results in the two Euklas conditions.

For the three tasks in the “difficult tasks” group, howev-er, the situation is different. “Euklas full” users had a larger advantage than participants in the other two groups. The code they used was longer and more complex (1828 LOC instead of 346 LOC), contained more errors (18 instead of

6) and the errors were more difficult to find (see Table 1). Therefore, we increased the maximum time that partici-pants were allowed to work on each task. The task comple-tion rates were higher for participants using “Euklas full” than participants using “Euklas lite” and participants in the control condition. Also, the results show that participants using “Euklas full” were able to fix more errors than partic-ipants in any of the other two conditions.

Overall, the results show that the “Euklas lite” version, which provided error detection features and standard quick fixes, already brought many improvements in comparison with Eclipse’s JavaScript editor, which did not offer such features. This is not surprising, since we know that high-lighting errors helps users. “Euklas full”, which offered additional analyses and additional quick fixes, improved performance even more. “Euklas full” especially showed advantages when the pieces of copied code were longer and/or more complex (as in Tasks 4-6). Euklas’s approach of considering a broader context for the computation of potential quick fixes has implications for many other pro-gramming languages, such as Java.

Using Eclipse’s marker feature for highlighting and fix-ing errors seems to be an appropriate UI choice as partici-pants had a very positive attitude towards this approach and considered it to be easy to learn. The most important aspect about using this feature is that participants knew what to expect from the provided quick fixes. They knew that these were usually right in “Euklas full”, but that they could sometimes also be wrong and might not be helpful. Such a situation was simulated in task 5, where participants had to fix a syntax error. The task also included a missing function definition that was not detected by either of the two Euklas versions. Participants were generally able to distinguish between these cases where they got standard “quick fixes” from Eclipse and where they got more sophisticated quick fixes from Euklas, since they did some manual checks to see if a proposed quick fix was appropriate or not before they used it.

There were a total of 24 errors that occurred after partic-ipants pasted the code from the source files into the target files. “Euklas full” provided 20 helpful suggestions for fixing these errors, plus 33 generic suggestions (e.g. declare a function with an empty body). Eclipse generally added in two additional suggestions per error that were not helpful at all. Participants sometimes picked one of Euklas’s generic fixes (e.g., for one of the missing functions in task 5). They knew that this was not sufficient for making the code work and therefore looked at the example code to manually find the correct piece of code. In cases where Euklas did not show an error at all (e.g., for one of the missing functions in task 5), participants did not perform worse than the par-ticipants in the control group, so there appears to be no disadvantage to using Euklas. However, the evaluation has some limitations. First, the evaluation only had the rather

19

small number of 4 participants in each condition. Second, participants used the tools for only one hour. Third, the tasks may not have been representative of realistic tasks. Although the pieces of code were real-world examples that were taken from the web, we had to reduce the complexity of dealing with these examples to limit the amount of time spent on completing each of the tasks.

6. Conclusion and Future Work In this paper, we presented our new Eclipse plug-in, Euklas, which supports JavaScript programmers in some of the tasks when using copy-and-paste-strategies for reusing example code. This kind of reuse does not create clones in a codebase [14], but instead helps users to reuse working code from others to introduce new functionality to their codebases and understand how this functionality can be implemented. Euklas supports these strategies by analyzing the target code for errors and by suggesting fixes for these errors. An important innovation is augmenting the fixes through an analysis of the code from where the example was copied. Our evaluation shows that Euklas’s users were able to fix a much higher number of copy-and-paste related errors than participants who used Eclipse’s JavaScript edi-tor, which does not provide any debugging support.

Euklas’s main contribution is analyzing the file from where code was copied to provide more detailed error de-scriptions and much better quick fixes for these errors. We think that applying Euklas’s ideas to editors used for other languages (e.g., Python, Java, and C++) could increase programmers’ performance in these situations in the same way it did in our case for JavaScript.

In addition to reducing the limitations discussed above, future work could include implementing some additional features for Euklas. One idea for providing improved error detections would be to design a heuristic for analyzing the context of a variable to try to determine its runtime type. This would allow the implementation of better quick fixes, since the system could distinguish between variables with the same name, but of a different type.

Euklas points to a future where programming support tools better help developers by taking into account all of the available contextual information, and the provenance of resources used. The success of Euklas shows that this ap-proach is feasible and can be successful, and developers can make effective use of recommendations, even when they are heuristic. The incorporation of extended copy and paste support into different kinds of editors would be a first step into this future.

Acknowledgements This work was conducted in 2010/2011, when the first author was a postdoc at Carnegie Mellon University. The

authors would like to thank Sara Kiesler for her invaluable assistance in the data analysis. The first author thanks the Alexander von Humboldt-Foundation for his Feodor Lynen Research Fellowship for Postdoctoral Researchers. This research has also been supported by SAP, Adobe and the National Science Foundation, under grant CCF-0811610. Any opinions, findings and conclusions or recommenda-tions expressed in this material are those of the author(s) and do not necessarily reflect those of the sponsors.

References [1] Lee, B. et al., Designing with interactive example galleries. in

CHI '10, 2257-2266. [2] Bajracharya, S. et al., C., Sourcerer: An internet-scale software

repository. in ICSE SUITE Workshop '09, IEEE, 1-4. [3] Brandt, J. et al., S.R., Example-centric programming: integrating web

search into the development environment. in CHI '10, 513-522. [4] Gross, P.A. et al., A code reuse interface for non-programmer middle

school students. in IUI '10, 219-228. [5] Hartmann, B. et al., What would other programmers do: suggesting

solutions to error messages. in CHI '10, 1019-1028. [6] Myers, B. et al., How Designers Design and Program Interactive

Behaviors. in VL/HCC'08, 177-184. [7] Brandt, J. et al., Two studies of opportunistic programming. in CHI

'09, 1589-1598. [8] Stylos, J. and Myers, B.A., Mica: A Programming Web-Search Aid.

in VL/HCC '06, 195-202. [9] Cottrell, R. et al., Semi-automating small-scale source code reuse via

structural correspondence. in FSE-16, 214-225. [10] Lim, S.C.S. and Lucas, P., JDA: a step towards large-scale reuse on

the web. in OOPSLA '06, 586-601. [11] Ko, A. J. et al., Eliciting Design Requirements for Maintenance-

Oriented IDEs. In ICSE '05, 126-135. [12] Kim, M. et al., An Ethnographic Study of Copy and Paste Program-

ming Practices in OOPL. in ISESE ‘04, 83-92. [13] Rosson, M.B. and Carroll, J.M. The reuse of uses in Smalltalk pro-

gramming. ACM TOCHI, 3 (3). 219-253. [14] Rahman, F., Bird, C. and Devanbu, P., Clones: What is that smell?.

In MSR ’10, 72-81. [15] Crockford, D., JavaScript: The Good Parts, Sebastopol, CA: O'Reilly

& Associates, 2008. [16] Lieberman, H., Paternò, F. and Wulf, V. End User Development.

Springer, Dordrecht, 2006. [17] Sahavechaphan, N. and Claypool, K., XSnippet: Mining For sample

code. in OOPSLA '06, 413-430. [18] Holmes, R., Walker, R.J. and Murphy, G.C. Approximate Structural

Context Matching: An Approach to Recommend Relevant Examples. IEEE TSE, 32 (12). 952-970.

[19] Hartmann, B. et al., Programming by a sample: rapidly creating web applications with d.mix. in UIST '07, 241-250.

[20] Jensen, S.H., Møller, A. and Thiemann, P., Type Analysis for JavaS-cript. in 16th International Symposium SAS, Springer, 238-255.

[21] Ayewah, N. et al., Using Static Analysis to Find Bugs. IEEE Soft-ware, 25 (5). 22-29.

[22] Ko, A. and Wobbrock, J., Cleanroom: Edit-Time Error Detection with the Uniqueness Heuristic. in VL/HCC '10, 7-14.

20