Slide 1

Experiment Framework

Results from the Working Groups

Revised Experiment Plan

People for Roles

Who signed up for what

Goals 2001/2 until next ISERN Meeting

Next steps

ISERN Distributed Experiment:

Action Session

Slide 2

Framework for the distributed experiment

• Industry Survey

– Descriptive analysis, no benchmark.

– Report characterizing types of responding organizations and their inspection processes.

• Benchmark an inspection technique

– Pick some inspection variant and study it across different contexts.

• Controlled experiment

– Get very accurate data about the improvement due to some inspection variant.

• Shadow Experiments: To lower risk before investing in Level 2 or Level 3 for industrial collaborators.

– ISERN will find a university course that would be willing to perform a first run of the same study.

Slide 3

Results Working Group Survey

• Discussion of Survey Goal

– Original Goal: Characterize context for inspection experiments (and collect some data also from organizations which do not currently use inspections)

– Input from Working Group: Also… ask survey respondents (who will be development team members rather than organizations) what they do for defect reduction early in development (as many development teams do not use formal inspections).

• Inspection survey questionnaires from Lund

– Were looked through and there were no objections to use them as basis for the development of survey instruments.

• Next steps

– Develop questionnaires for context variables of experiment

– Option to plan a larger survey on early defect reduction approaches (based on experiment context survey?).

Slide 4

Goals for the technique to be benchmarked

• Require evidence of its likely benefit

• Should be widely applicable (maximize potential pool of participants)

• Some version should be able to be taught in a “reasonable” time

• Should be of genuine interest to target audience

• Results should be actionable

Slide 5

Results Working Group Experiment Design

• Decision on experiment topic:– Design Inspection (UML?)

• Target business areas for experiment:– Embedded systems, banking, insurance, …

• Catch interest: package as training– Possibly use situated learning? – 2 days– Available on web/streaming video– Pricing policy?– Training changes the document to be inspected?

• Ethical issues: need common NDA

• Experiences so far with design inspection

• Material available from past experiments

Slide 6

Who signed up for what?

• Sign-up status:

– Survey (without experiment): 12 X, 3 ?

– Experiment (with context survey): 6 X, 11 ?

– Shadow experiment: 11 X, 2 ?

• If you have not signed up, and want to do so:

– Stefan Biffl has the list.

Slide 7

Organization: People for roles

• Librarian: Marcus Ciolkowski.

• Working group chairs

– Experimentation design: Forrest Shull

– Survey design: Ross Jefferey

– Meta-analysis: ?? (plans data analysis)

• Steering Committee: contributors to experiment

• Experiment Preparation Committee: authors of experiment documentation

• Experiment Coordinator: knows experiment status

• QA for Data Repository: checks incoming data

Slide 8

Potential Steering Committee Members

• OORT experiments

– Shull, Travassos, Conradi

• Inspection

– Miller, Wohlin, Maldonado, Succi

– Wood, Dunsmore, Karlsson

– Biffl, Ciolkowski

• Esernet

• If you want to take part in the Steering Committee, talk to Stefan Biffl.

Slide 9

Goals and Next Steps

• Next steps

– Make package of currently available material for design inspection (in English, for translation by local researchers as they see fit).

– Send package out to experiment group for review.

• Time table

– End of September 2001: 1st version of package and survey instrument.

– X-mas 2001: Final version of package and survey instrument.

• Goals for 2001/2 until next ISERN Meeting

– Conduct and evaluate survey with inspection focus.

– Conduct some experiments.

– Conduct meta-analysis on experiment results and data from past experiments.

Slide 10

Server for Discussion

• http://asercweb.ee.ualberta.ca:8000/ISERN

• Login: name

• Password: Lastname