[13 - B] Experiment reporting

‹#› Het begint met een idee

Experiment reporting

Ivano Malavolta

Vrije Universiteit Amsterdam

2

Roadmap

● Experiment reporting

● Examples of empirical studies in SE

Ivano Malavolta / S2 group / Experiment reporting and recommendations


3

Let’s take a step back



19 Ivano Malavolta / S2 group / Empirical software engineering

5 - Presentation & package

● Document the results

● Prepare replication package

● Sum up lessons learned

● Write down reflectionsExperiment

scoping

Experiment planning

Idea

Experiment operation

Analysis & interpretation

Presentation & package


4 Ivano Malavolta / S2 group / Experiment reporting and recommendations

Basic structure of an experiment report

(Structured) Abstract

Introduction/Motivation

(Related Work)

Experiment design

Experiment execution

Data analysis

Threats to validity

Discussion

Conclusions

References


5

Abstract

● People judge papers by their abstracts○ usually decide whether to read the whole paper based on abstract

● It's important for the abstract to tell the whole story○ background and context

○ goals

○ method

○ results

○ conclusion

● Often it is the only part freely accessible



6

Mini-lab

1. Read the 3 abstracts

2. Rank them according to how they provide insights about:

a. problem being addressed

b. obtained results and main contributions

c. type of evaluation/empirical method

3. Which abstract is easier for a secondary study?



7

The structured abstract

● Context

○ brief explanation of the motivation for conducting the study

● Objective

○ aim, objects, focus, perspective of the study (rephrase GQM)

● Method

○ experimental design, number and kind of objects/subjects, selection

criteria, data collection and analysis procedures

● Results

○ main findings

● Conclusion

○ impact of the obtained results



8

Example of structured abstract


9

Introduction and motivation

● Introduction

○ Mini-version of the whole paper

■ Attacked problem■ Proposed solution or experiment■ Main results■ Main contributions (how to “use the paper”)■ Structure of the paper

● Motivation:

○ Scope of the work○ Problem statement○ Research objectives○ encourages to read the paper



10

Related work

● Why this paper exists?

● Complete picture about:

○ experiments with similar goals○ experiments with similar objects/subjects○ papers with similar goals, but not empirical → they lack in provided

evidence

● Two main strategies:

○ Paper-by-paper comparison○ Catalogue of related papers + overall comparison



11

Experiment design and execution

● Design: rewalk through all the experiment plan

○ GQM○ hypotheses○ objects and subjects○ variables ○ measurements○ experiment design○ data analysis strategies (e.g., used statistical tests)

● Execution: how the experiment plan has been put in practice

○ preparation (eg code instrumentation)○ data collection procedure○ data clean up○ any deviation from the plan○ reference to replication package


12

Analysis and threats to validity


● Analysis (or Results)

○ Demographics○ Descriptive statistics and data exploration○ For each research question:■ hypothesis testing■ discussion of effect size■ quick elaboration on obtained results

● Threats to validity○ Internal○ External○ Construct○ Conclusion


13

Discussion and Conclusions


● Discussion

○ Elaborate on the obtained results○ Impact into practice (industry)○ Make the results actionable○ Lessons learnt

● Conclusions○ Summary of the main contributions of the paper○ Future work■ other experiments on different aspects■ better ways to perform the experiment■ ...


14 Ivano Malavolta / S2 group / Experiment reporting and recommendations

More details? You have a full paper about experiment reporting!

Chapter 8


15

Replication package

3. Figures

○ exploratory figures

○ final figures

You have to follow this structure when providing the replication package of your final assignment!

1. Data

○ raw data (usually CSV)

○ processed data (CSV)

○ (name it properly!)

2. Source code (commented!)

○ measurement scripts (Python)

○ data processing scripts (R)

○ analysis scripts (R)

4. Text

○ README file

○ (optional) experiment notes


16

Examples of empirical studies in SE

● Nejati, Javad, and Aruna Balasubramanian. "An in-depth study of mobile browserperformance." Proceedings of the 25th International Conference on World WideWeb. International World Wide Web Conferences Steering Committee, 2016.

● Hassan, Safwat, Weiyi Shang, and Ahmed E. Hassan. "An empirical study ofemergency updates for top android mobile apps." Empirical Software Engineering(2016): 1-42.

● Tufano, Michele, et al. "An empirical investigation into the nature of test smells."Proceedings of the 31st IEEE/ACM International Conference on AutomatedSoftware Engineering. ACM, 2016.

● Linares-Vásquez, Mario, et al. "Mining energy-greedy api usage patterns in androidapps: an empirical study." Proceedings of the 11th Working Conference on MiningSoftware Repositories. ACM, 2014.

● Procaccianti, Giuseppe, Héctor Fernández, and Patricia Lago. "Empiricalevaluation of two best practices for energy-efficient software development."Journal of Systems and Software 117 (2016): 185-198.



17

What this lecture means to you?

● You know how to present your experiment

○ push for clarity, soundness, and completeness

● Replication package needed

○ helps community building

○ enables replications

○ gives you also visibility!

● Examples of very good papers

○ Be inspired!



18

Readings

Chapter 11 Chapter 16 Chapter 8



Next lecture

Guest lecture by Fabio Palomba

○ VU Green Office http://www.greenofficevu.nl/

○ Room HG KC-18

○ Thursday October 19th, 11 AM

TOPIC: Information retrieval for code changes based on user

reviews of mobile apps


Next steps

Vrije Universiteit Amsterdam8

Courseschedule

3 types of lectures:

• Theory

• Tutorial

• Hands-on session

Vrije Universiteit Amsterdam14

Scheduleanddeliverables

• Deliverable 1 (20% of the final grade)

• Experiment goal and scope description

• Deliverable:

• written report

• Deadline: 21 September: 23:59


• Detailed design of the experiment

• Deliverable:

• written report

• Deadline: 5 October: 23:59


• Final report of the experiment

• Deliverable:

• written report

• experiment execution scripts and source code

• raw data

• analysis scripts in R

• Deadline: 29 October: 23:59

Technology

[13 - B] Experiment reporting