PROMISE 2008: wrap up
The PROMISE effect:
Do more together than separately
Gary D. Boetticher Tim Menzies Tom Ostrand Guenther Ruhe
2
The other side of PROMISE
PROMISE is not just an annual meeting, it is community.
How much contact have you had with your fellow PROMISE-ians during the year?
Are you exploiting the “PROMISE effect”? Do more together than separately.
3
Case study #1
Cukic & Menzies: new NSF award 2008 to 2011 “Overcoming ceiling effects in defect predictors
Surprisingly good reviews High praise for the connection to the PROMISE repository What are the broader impacts of the proposed activity?
“… The work has significant broader impacts and the PROMISE repository that the PIs are a part of provides them a platform for making further contributions to the SE community. …”
Summary Statement “… The PROMISE repository and the work of the PI represents a
significant input to the proposal. …. “ So the repository is now a tool for access funding.
4
Case study #2 For several years, Ostrand, Weyuker, Bell at AT&T held the
state of the art in defect localization (predicting the subset of the code with most errors)
2007: At PROMISE’07: Koru argues that defect distribution is logarithmic (I.e. more errors in the
smaller modules) Sept 2008:
Milton (WVU) tells Menzies that their new defect localizer is failing. Menzies says “try crawling the suspect modules in the Koru order”
(smallest modules first) Jan’08:
Milton shows the new localizer dramatically out-performs standard methods
Mar’08: Milton hosted at AT&T
for a week by Ostrand Tests the new localizer inside the firewalls at AT&T
5
Case study #3
TSE 2007, May, Kitchenham et.al. assess
within-company (WC) data vs cross-company (CC) data for effort prediction. August ‘07 : Menzies (USA) realizes that the PROMISE repository could
perform the same study, but for defect prediction. Crowd sources the PROMISE community for collaborators
Sept ‘07: Bener&Turhan (Turkey) run the experiments Turkey makes the data, USA runs the statistics CC results: best PD results ever seen with this data (but very bad PF rates)
Oct ‘07: Submission to TSE by Menzies, Bener, Turhan CC depreciated for defect prediction
Feb 14: TSE decision: revise & resubmit May 1: Resubmission with 3 news data sets (Turkish whitegoods)
Paper now much stronger PROMISE repository now has 3 more data sets
Other Bener, Menzies, Turhan papers: two Defects’08, ASE’08
6
Case study 4,5,6….
Over to you… Any other stories to add of interactions
between PROMISE-ians?
7
The PROMISE effect
Do more together separately. Look around the room
What ideas are in the air? What is the value-added of the person sitting next
to you? What are you plans for collaboration over the next
7 months (in time for the next PROMISE cfp)? And we can use web-tools to foster this
community: Chat rooms, discussion groups, wikis, code library
8
Proposal: a PROMISE chat room
Two minutes chatting can sometimes beat 2 months of reading
?Meebo.com Web-based,
platform independent
9
Proposal:a PROMISE discussion list Chatting is a hardly
permanent record of a discussion
Good to access what others have said in the past
?groups.google.com
10
Proposal: a PROMISE wiki
Promisedata.org is read only (except for comments) Good for workshop admin
and data storage Chat rooms and discussion
groups good for discussion “volleys”
Wikis are better for summarizing and synthesizing on-going discussions write, rewrite access
?wikispaces.org
11
Proposal:a shared PROMISE code library Reading is one thing Running code is
another. Have a shared
open-source, code base, complete with make files
Able to reproduce prior results with one command.
?code.google.com ?a Debian package
12
Proposal:the PROMISE’09 awards Source award
Awarded to the individuals or groups that contribute the most new data, with baseline results, to the repository.
Service award Awarded to the individuals or groups that offer most
support to others in the new groups/ chat room/ wiki.
Science award Awarded to the 2009 paper that best repeats/extends/
interacts with prior results and/or authors of a past PROMISE paper.