On the Value of User Preferences in Search-Based Software Engineering

On the Value of User Preferences in Search-Based Software Engineering:

A Case Study in Software Product Lines

Abdel Salam Sayyad Tim MenziesHany Ammar

West Virginia University, USA

International Conference on Software EngineeringMay 23rd, 2013

2

Sound bites

Feature-based SERepresentation and validation: importantMulti-objective Configuration: interesting

Stop tinkering with small stuff!Minor improvements welcome…

… but not insightful!

Enough with the usual suspects: NSGA-II, SPEA2, etc.

If user preferences matterThen the best optimizer understands preferences the best

Roadmap

① Feature-based SE② Evolutionary Algorithms③ Experiment & Results④ Conclusion

Roadmap

① Feature-based SE② Evolutionary Algorithms③ Experiment & Results④ Conclusion

5

Welcome to the new world!Old world:

product-based SE

e.g. Microsoft office

New world: app-based SE

e.g. Apple app store

6

The times, they are a changin’

Old world: product-based SEe.g. Microsoft office • Vendors tried to retain their

user base via some complete ecologies

• One software solution for all user needs (e.g. Microsoft Office).

• Large, complex, software platforms, – very slow to change.

7

The times, they are a changin’

New worlde: app-based SE • Smart phones and tablet-

based software

• Users choosing many numbers of small apps from different vendors, – each performing a specific

small task.

• Vendors must quickly and continually reconfigure apps – To retain and extend their

customer base.

e.g. Apple app store

8

Feature–oriented domain analysis [Kang 1990]

• Feature models = a lightweight method for defining a space of options

• De facto standard for modeling variability

Cross-Tree Constraints

Cross-Tree Constraints

9

What are the user preferences?• Multi-objective optimization = navigating competing concerns

– Success criteria = choose features that achieve the user preferences!• Suppose each feature had the following metrics:

1. Boolean USED_BEFORE?2. Integer DEFECTS3. Real COST

• Show me the space of “best options” according to the objectives:1. That satisfies most domain constraints (0 ≤ #violations ≤ 100%)2. That offers most features3. That we have used most before4. Using features with least known defects5. Using features with least cost

10

Diving deeper• Much prior work in SBSE (*)

--------------------------(*) Sayyad and Ammar, RAISE’13

explored tiny objective spaces (2 or 3 objectives)

Used NSGA-II

Didn’t state why!

Roadmap

① Feature-based SE② Evolutionary Algorithms③ Experiment & Results④ Conclusion

12

Multi-objective Optimization

Higher-level Decision Making

The Pareto Front

The Chosen Solution

13

Survival of the fittest(according to NSGA-II [Deb et al. 2002])

Boolean dominance (x Dominates y, or does not):- In no objective is x worse than y- In at least one objective, x is better than y Crowd

pruning

14

Tinkering with small stuff…

ssNSGA-II: steady state [Durillo ‘09]

Other Algorithms: Ranking criteria:NSGA-II

SPEA2 [Zitzler ‘01] More focus on diversity, with a new diversity measure.

FastPGA [Eskandari ‘07]Borrowed criteria from NSGA-II, and diversity measure from SPEA2

MOCell (Cellular GA) [Nebro ‘09] NSGA-II

NSGA-IIMOCHC (re-designed selection, crossover, and mutation) [Nebro ‘07]

Boolean Dominance

Boolean Dominance

Boolean Dominance

Boolean Dominance

Boolean Dominance

15

Indicator-Based Evolutionary Algorithm (IBEA) [Zitzler and Kunzli ‘04]

1) For {old generation + new generation} do– Add up every individual’s amount of dominance with

respect to everyone else

– Sort all instances by F– Delete worst, recalculate, delete worst, recalculate, …

2) Then, standard GA (cross-over, mutation) on the survivors Create a new generation Back to 1.

Continuous Dominance

16

An important property of IBEA

• Allows solutions to crowd close to the zero point.

• In our problem: Allows many solutions that achieve violations equal or close to zero… then allows optimization in the other 4 dimensions…

• Crowd pruning can hurt!

• Using real-valued benchmark functions, Wagner et al. (EMO 2007) confirm that the performance of NSGA-II and SPEA2 rapidly deteriorates with increasing dimension, whereas indicator-based algorithms (e.g. IBEA) cope very well with high-dimensional objective spaces.

Roadmap

① Feature-based SE② Evolutionary Algorithms③ Experiment & Results④ Conclusion

18

2 Case studies from SPLOT

Cross-treeconstraints

Algorithms

• 7 Algorithms from jMetal: http://jmetal.sourceforge.net/ – IBEA, NSGA-II, ssNSGA-II, SPEA2, FastPGA, MOCell, MOCHC.

19

• Features are bits in the “product” string.

• Same settings for all algorithms.

20

4 studies:Two, three, four, five- objectives

The five objectives:1. satisfy most domain constraints (0 ≤ violations ≤ 100%)2. offer most features3. most used before4. least known defects5. least cost

Two objectives = 1,2Three objectives = 1,2,5Four objectives = 1,2,3,5Five objectives = 1,2,3,4,5

21

HV = hypervolume of dominated regionSpread = coverage of frontier% correct = % of solutions that fully satisfy the constraints

22Comment 1: all about the same for the 2-objective problem

HV = hypervolume of dominated regionSpread = coverage of frontier% correct = % of solutions that fully satisfy the constraints

23Comment 2: IBEA is better on HV.

HV = hypervolume of dominated regionSpread = coverage of frontier% correct = % of solutions that fully satisfy the constraints

24Comment 3: IBEA has no spread operators, but gets best spread

HV = hypervolume of dominated regionSpread = coverage of frontier% correct = % of solutions that fully satisfy the constraints

25Comment 4: All the non-IBEA algorithms are very similar

HV = hypervolume of dominated regionSpread = coverage of frontier% correct = % of solutions that fully satisfy the constraints

26

HV = hypervolume of dominated regionSpread = coverage of frontier% correct = % of solutions that fully satisfy the constraints

Comment 5: IBEA does much, much better on constraints

27

HV = hypervolume of dominated regionSpread = coverage of frontier% correct = % of solutions that fully satisfy the constraints

Comment 6: IBEA’s advantage is more striking with E-Shop

28

Why is this interesting?

Other Algorithms• The usual suspects are widely,

uncritically used in many applications– E.g. especially NSGA-II and SPEA2

• Focused on internal algorithmic tricks– Techniques for

• improving spread • Improving HV• Avoid overlaps in cross-over of

dominated space• etc.

• And the net effect of all those differences?– Not much

IBEA• Rather stupid on those

internal tricks– Just does same old crossover

mutate GA– Plus: aggressive exploration

of the preference space

• And the net effect of all those differences– Better spreads– Better HV– Fewer constraint violations

Conclusion: preference is power

Roadmap

① Feature-based SE② Evolutionary Algorithms③ Experiment & Results④ Conclusion

30

Current Achievements and Future Work

• ICSE’13: initial results proving IBEA’s advantage.• CMSBSE’13: IBEA runtime improvement.• (Under review): – mutation operator tailored to feature tree. 2-3

orders-of-magnitude improvement in runtime.– Scalability to 6000+ real feature model using

innovative “population seeding” method.

• Future Work:– Further scalability.– Interactive configuration.

31

Sound bitesThe new age of the app

In this new world: use FODA(feature-oriented domain analysis)

Stop tinkering with small stuff Many MEOAs have strikingly similar performance

Enough with the usual suspects: NSGA-II, SPEA2, etc.Too much uncritical application of these algorithms

If user preferences matterThen the best optimizer understands preferences the bestIBEA: aggressive preference exploration

AcknowledgmentThis research work was

funded by the Qatar National Research Fund

(QNRF) under the National Priorities Research Program

(NPRP) Grant No.: 09-1205-2-470.