NOTICE: A Framework for Non-functional Testing of Compilers

Mohamed BOUSSAA

OlivierBARAIS

GersonSUNYE

BenoitBAUDRY

2016 IEEE International Conference on Software Quality, Reliability & Security (QRS 2016)

August 1-3, 2016 - Vienna, Austria

INRIA Rennes, France

2016 IEEE International Conference on Software Quality, Reliability & Security (QRS 2016)

August 1-3, 2016 - Vienna, Austria1

a1. Context

a2. Motivating Example

a3. NOTICE: A Framework for Non-functional Testing of Compilers

a4. Performance Evaluation

a5. Conclusion

Outline

2

Motivation

C Compilers

Source code Machine code

Machine code

Machine code

Optimizations

Current innovations in science and industry demand ever-increasing computingresources while placing strict requirements on system performance, power

consumption, size, response, reliability, portability and design time

Satisfy the non-functional requirements

for a broad range of programs and

architectures

Resource constraints

3

Compiler fine/auto-tuning is complex

4

Huge design space for optimization options (more than 150 optimizations)

• compiling a program means trading off between various objectives • compilation time, code quality, code size, ...

Constructing a good set of optimization levels (-Ox) is hard

• conflicting objectives, complex interactions, unknown effect of some optimizations, ...

4

Trying to please everyone

5

Program-independent universal sequences (e.g., -O1, -O2, -O3, etc.)

Based on heuristics and experience

Each optimization level allows trading off various objectives

• O1: "take your time, give it your best shot"• O2: "optimize, and be quick about it"• O3: "I’m feeling lucky, and have lots of time"

How efficient are predefined/universal compiler levels?

5

Motivating Example

GCC 4.8.4:- 78 optimizations - 278 combinations

6

Speedup,Memory,

etc.

ResourceConstraints

WHY ALWAYS

ME !!????

- Testing each optimization configuration is impossible

-BOSS: Clients complain about the high memory consumption

-BOSS: Is it possible to consume less CPU? we don’t have enough resources/money

-BOSS: Please, can we optimize even more ?

Good luck Son !!

- Heuristics are needed

6

NOTICE: A Framework for Non-functional

Testing of Compilershttps://noticegcc.wordpress.com

7

Contributions

1- Diversity-based exploration

Novel formulation of the compiler optimization problem using Novelty Search Diverse optimization sequences Explore the large search space by considering Novelty as the main objective

2- Microservice-based infrastructure

Execute and monitor of the different variants of optimized code using system containers

Resource isolation and management Provide a fine-grained understanding and analysis of compilers behavior regarding

optimizations Automatic extraction of non-functional properties relative to resource usage Finely auto-tuning compilers according to user (non-functional) requirements

We propose:

8

Diversity-based exploration

gcc –c test.c –fno-dce –fno-dse –fdce -fno-align-loops …

Mutation:

Crossover:

Best solution Solution with best non-functional improvement

0 0 1 0 …

Step 2: Evaluation

… Archive:

Novelty metric:

Step 3: Selection

Step 4: Evolutionary

operators 0 1 1 1 0 …

0 1 1 1 0 …

1 0 0 1 1 …

Go To Step 2

Solution representation:

Saves solutions that get a novelty metric value higher than a specific novelty threshold value.

Calculate the distance of one solution from its K Nearest neighbors in current population and in the Archive.

Step 1: Random

generation

9

Select solutions to evolve based on novelty scores.

Tournament selection:

Contributions

1- Diversity-based exploration

Novel formulation of the compiler optimization problem using Novelty Search Diverse optimization sequences Explore the large search space by considering Novelty as the main objective

2- Microservice-based infrastructure

Execute and monitor of the different variants of optimized code using lightweight system containers

Provide a fine-grained understanding and analysis of compilers behavior regarding optimizations

Resource isolation and management Automatic extraction of non-functional properties relative to resource usage Finely auto-tuning compilers according to user (non-functional) requirements

We propose:

10

NOTICE Infrastructure

000

000

NOTICE

Compile and execute optimized code within a new

container instance

Gather at runtime non-functional properties of running programs under test

Save information relative to resource consumptions within a times series database

Analysis of the performance and non-functional properties

of programs under test

1

2

3

4

Code Execution

RuntimeMonitoring

Time seriesDatabase

PerformanceAnalysis

11

NOTICE Infrastructure

OptimizationsComponentUnder Test

MonitoringComponent

Back-endDatabase

Component

Cgroup file systemsRunning…

Monitoring records

Front-endVisualizationComponent

Time-series database

HTTP Requests

CPU

Memory

…

12

Evaluationhttps://noticegcc.wordpress.com/experimental-results/

13

Experimental Setup

v4.8.4

Random C code generator

For monitoring

For storage

Optimizations

Mono ObjectiveNovelty Search (NS)Genetic Algo (GA)

Random Search (RS)

Multi ObjectiveNovelty Search (NS-II)

NSGA-II

Speedup (S)

Meta-heuristics

Program under

test

Compiler

Algorithm parameters

Evaluation metrics

Memory consumption reduction (MR)

CPU consumption reduction (CR)

Over -O0

Trade-off <execution time - memory usage>

14

Research QuestionsRQ1: Mono-objective SBSE Validation.

Optimizations

Non-functionalmetric

Training set programs

Best sequence

RQ2: Sensitivity of input programs to optimization sequences

Unseen programs

Non-functionalimprovementBest sequence

in RQ1

RQ3: Impact of speedup on resource consumption.

RQ4: Trade-offs between non-functional properties.

Best SpeedupSequence

In RQ1

Impact on resource

consumption OptimizationsPareto front

solutions

15

Training set programs

Multi-objective search

Mono-objective search

Non-functionalTrade-off

<time-memory>

Input program

RQ1- ResultsRQ1: Mono-objective SBSE Validation.- Training set: 10 Csmith programs- Average S, MR, and CR- Comparison: Ox, RS, GA and NS

Key findings for RQ1:– Best discovered optimization sequences using mono-objective search techniques always provide better results than standard GCC optimization levels.– Novelty Search is a good candidate to improve code in terms of non-functional properties since it is able to discover optimization combinations that outperform RS and GA.

Search for best optimization sequence

Best sequence

Optimizations

Non-functionalMetric

Training set programs

16

RQ2- Results

Key findings for RQ2:– It is possible to build general optimization sequences that perform better than standard optimization levels– Best discovered sequences in RQ1 can be mostly used to improve the memory and CPU consumption of Csmith programs. To answer RQ2, Csmith programs are sensitive to compiler optimizations.

RQ2: Sensitivity.- 100 unseen Csmith programs- O2 vs O3 vs NS

Unseen programs

Non-functionalimprovement

Best SequenceIn RQ1

17

RQ3- ResultsRQ3: Impact of optimizations on resource consumption.- Ox vs RS vs GA vs NS

Key findings for RQ3: – Optimizing software performance can induce undesirable effects on system resources.– A trade-off is needed to find a correlation between software performance and resource usage.

Best SpeedupSequence

In RQ1 Training set programs

Impact on Resource CPU & memory

18

Memory reduction

Increase of resource usage

CPU reduction

RQ4- ResultsRQ4: Trade-offs between non-functional properties. - 1 Csmith program- Trade-off <execution time-memory usage>

Key findings for RQ4: – NOTICE is able to construct optimization levels that represent optimal trade-offs between non-functional properties.– NS is more effective when it is applied for mono-objective search.– NSGA-II performs better than our NS adaptation for multi-objective optimization. However, NS-II performs clearly better than standard GCC optimizations and previously discovered sequences in RQ1.

19

Optimizations Pareto frontsolutions

Multi-objective searchTrade-off time/memory

Input program

Pareto front NS-II(multi-objective)

Ofast O3 O2

O1

Best CPU reduction (mono-

objective)

Best memory reduction(mono-objective)

Pareto front NSGA-II(multi-objective)

Conclusion

20

Conclusion

21

Novel formulation of the compiler optimization problem based on Novelty Search

Novelty Search is able to generate effective optimizations

Automated tool for automatic extraction of non-functional properties of optimized code

Automatically extract information about memory and CPU consumption

Summary

Explore more trade-offs amongresource usage metrics

Evaluate NOTICE:• on real world benchmarks• other case studies (i.e.,

compilers, programs, etc)

Future directions

21

https://noticegcc.wordpress.com/ 22

Questions?

Additional slides

23

Tool Support

24

Functional Testing of Compilers

PLDI’11

PLDI’14

Literature Overview

ICSE’16

25

Non-Functional Testing of Compilers

Literature Overview

CGO’08

ACSAC’08

PLDI’04

26

Prior work is insufficient

Testing the non-functional properities pose several new challenges:

- Different cost-benefit trade-offs (e.g., Speedup/memory or CPU usage)

- Finely auto-tuning compilers according to user (non-functional) requirements

- Performance is the major concern (e.g., speedup)

- Ignore other important non-functional properties (e.g., resource consumption properties)

- Evaluation is based on a small set of input programs (e.g., Spec CPU benchmarks)

27

Given a set of compiler optimization options {F1, F2, ..., Fn}, How can we find

the combination that maximize program performance better than standard

optimization levels ?

Do this efficiently, without the use of a priori knowledge of the optimizations and their interactions

From

to

From

to

Problem Statement

28

NSGA-II overview

• NSGA-II: Non-dominated Sorting Genetic Algorithm (K. Deb et al., ’02)

Parent Population

Offspring Population

Non-dominated sorting

F1

F2

F3

F4Crowding distance sorting

Population in next generation

MOEA Framework http://moeaframework.org/

29

NSGA-II overview

30