Google ROADEF/EURO challenge 2011-2012:

Machine reassignment

1 Problem description

The aim of this challenge is to improve the usage of a set of machines. A machinehas several resources as for example RAM and CPU, and runs processes whichconsume these resources. Initially each process is assigned to a machine. Inorder to improve machine usage, processes can be moved from one machine toanother. Possible moves are limited by hard constraints, as for example resourcecapacity constraints, and have a cost. A solution to this problem is a newprocess-machine assignment which satisfies all hard constraints and minimizesa given objective cost.

1.1 Decision variables

Let M be the set of machines, and P the set of processes. A solution is anassignment of each process p P to one and only one machine m M; thisassignment is noted by the mapping M(p) = m in this document. The originalassignment of process p is denoted M0(p). Note the original assignment isfeasible, i.e. all hard constraints are satisfied.

For instance, if M = {m1,m2} and P = {p1, p2, p3}, then M(p1) = m1,M(p2) = m1, M(p3) = m2 means processes p1 and p2 run on machine m1 andprocess p3 runs on machine m2.

1.2 Hard constraints

1.2.1 Capacity constraints

Let R be the set of resources which is common to all the machines, C(m, r) thecapacity of resource r R for machine m M and R(p, r) the requirement ofresource r R for process p P. Then, given an assignment M , the usage Uof a machine m for a resource r is defined as:

U(m, r) =

pP such thatM(p)=m

R(p, r)

1

A process can run on a machine if and only if the machine has enoughavailable capacity on every resource. More formally, a feasible assignment mustsatisfy the capacity constraints:

m M, r R, U(m, r) C(m, r)

Consider for example machines M = {m1,m2}, processes P = {p1, p2, p3}and resources R = {CPU,RAM}. Available CPU is C(m1, CPU) = 16,C(m2, CPU) = 8 and available RAM is C(m1, RAM) = 16, C(m2, RAM) = 4.CPU requirements are R(p1, CPU) = 6, R(p2, CPU) = 1 and R(p3, CPU) = 3.RAM requirements are R(p1, RAM) = 13, R(p2, RAM) = 3 and R(p3, RAM) =1.

Assignments M(p1) = m1, M(p2) = m1, M(p3) = m2 and M(p1) = m1,M(p2) = m2, M(p3) = m2 satisfy capacity constraints.

However assignment M(p1) = m2, M(p2) = m1, M(p3) = m2 is not feasibleas m2 has not enough available CPU . In the same way, assignment M(p1) = m1,M(p2) = m1, M(p3) = m1 is not feasible as m1 has not enough available RAM .

1.2.2 Conflict constraints

Processes are partitioned into services. Let S be a set of services. A services S is a set of processes which must run on distinct machines. Note that allservices are disjoint.

s S, (pi, pj) s2, pi 6= pj M(pi) 6= M(pj)

For instance M = {m1,m2,m3,m4}, P = {p1, p2, p3}, S = {sa, sb} withsa = {p1} and sb = {p2, p3}, and assignments M0(p1) = m1, M0(p2) = m1 andM0(p3) = m3. Process p1 can be reassigned to any machine, e.g. M(p1) = m2,M(p2) = m1 and M(p3) = m3. However p2 cannot be reassigned to machinem3, i.e. M(p1) = m1, M(p2) = m3 and M(p3) = m3, because process p3 is aprocess of service sb too and is currently running on m3.

1.2.3 Spread constraints

Let L be the set of locations, a location l L being a set of machines. Notethat locations are disjoint sets. For each s S let spreadMin(s) N be theminimum number of distinct locations where at least one process of service sshould run. The constraints are defined by:

s S,lL

min

(1,

{p s | M(p) l}) spreadMin(s)For instance M = {m1,m2,m3,m4}, P = {p1, p2}, S = {s} with s =

{p1, p2}, L = {{m1,m2}, {m3}, {m4}}, M0(p1) = m1, M0(p2) = m3 andspreadMin(s) = 2. Process p1 can be reassigned to m2 or m4. Process p2 canbe reassigned to m4. But to satisfy the spread constraint, process p1 cannot be

2

reassigned to m3, and p2 cannot be reassigned to m1 or m2 as in these casesonly one location runs s processes.

1.2.4 Dependency constraints

Let N be the set of neighborhoods, a neighborhood n N being a set ofmachines. Note that neighborhoods are disjoint sets.

If service sa depends on service sb, then each process of sa should run in theneighborhood of a sb process:

pa sa, pb sb and n N such that M(pa) n and M(pb) n

Note dependency constraints are not symmetric, i.e. service sa depends onservice sb is not equivalent to service sb depends on service sa.

Consider for instanceM = {m1,m2,m3,m4}, P = {p1, p2, p3}, S = {sa, sb}with sa = {p1} and sb = {p2, p3}, initial assignments M0(p1) = m1, M0(p2) =m1 and M0(p3) = m3, and neighborhood N = {{m1}, {m2}, {m3,m4}}. If sadepends on sb, p1 can be reassigned to m3 or m4 as p3 is a process of service s

b

and runs in the {m3,m4} neighborhood. However p1 cannot be reassigned tom2 as there is no s

b process running in the neighborhood of m2. In the sameway, process p2 cannot be reassigned to any other machine as p1 needs a s

b

process in its neighborhood.

1.2.5 Transient usage constraints

When a process p is moved from one machine m to another machine m someresources are consumed twice; for example disk space is not available on machinem during a copy from machine m to m, and m should obviously have enoughavailable disk space for the copy. Let T R R be the subset of resources whichneed transient usage, i.e. require capacity on both original assignment M0(p)and current assignment M(p). Then the transient usage constraints are:

m M, r T R,

pP such thatM0(p)=m M(p)=m

R(p, r) C(m, r)

Note there is no time dimension in this problem, i.e. all moves are assumedto be done at the exact same time. Then for resources in T R this constraintsubsumes the capacity constraint.

For instance M = {m1,m2,m3} and P = {p1, p2}, M0(p1) = m1 andM0(p2) = m2. R = {CPU,DISK} and T R = {DISK}. C(m1, CPU) = 3,C(m2, CPU) = 3, C(m3, CPU) = 3, C(m1, DISK) = 10, C(m2, DISK) = 10,C(m3, DISK) = 7, R(p1, CPU) = 1, R(p2, CPU) = 1 and R(p1, DISK) = 8,R(p2, DISK) = 6.

Lets suppose process p2 was moved from m2 to m3, so M(p2) = m3 andM0(p2) = m2. Process p1 cannot be moved from m1 to m2 even if no process iscurrently running on machine m2. This is due to the transient usage constraintwhich in some way still consumes 6 DISK on machine m2.

3

1.3 Objectives

The aim is to improve the usage of a set of machines. To do so a total objectivecost is built by combining a load cost, a balance cost and several move costs.

1.3.1 Load cost

Let SC(m, r) be the safety capacity of a resource r R on a machine m M.The load cost is defined per resource and corresponds to the used capacity abovethe safety capacity; more formally:

loadCost(r) =mM

max(0, U(m, r) SC(m, r)

)For instance M = {m1,m2}, P = {p1, p2} and R = {r}, M0(p1) = m1,

M0(p2) = m1, C(m1, r) = 100, C(m2, r) = 100, SC(m1, r) = 10, SC(m2, r) =50, R(p1, r) = 7 and R(p2, r) = 12. Then loadCost(r) = max(0, 12+710) = 9.Moving process p2 from machine m1 to machine m2, reduces the load cost from9 to 0, i.e. loadCost(r) = max(0, 7 10) + max(0, 12 50) = 0.

1.3.2 Balance cost

As having available CPU resource without having available RAM resource isuseless for future assignments, one objective of this problem is to balance avail-able resources. The idea is to achieve a given target on the available ratio oftwo different resources. Let B be a set of triples defined in NR2. For a giventriple b = r1, r2, target B, the balance cost is:

balanceCost(b) =mM

max(0, target A(m, r1)A(m, r2)

)with A(m, r) = C(m, r) U(m, r)

For instance M = {m1,m2}, P = {p1, p2, p3} and M0(p1) = m1, M0(p2) =m1, M0(p3) = m2. R = {CPU,RAM}, available CPU is C(m1, CPU) = 16,C(m2, CPU) = 8 and available RAM is C(m1, RAM) = 16, C(m2, RAM) =12. CPU requirements are R(p1, CPU) = 2, R(p2, CPU) = 8 and R(p3, CPU) =5. RAM requirements are R(p1, RAM) = 8, R(p2, RAM) = 1 and R(p3, RAM) =1. Machines should be balanced such that for one unit of available CPU , 2 unitsof RAM are available, i.e. B = {CPU,RAM, 2}.

Then the balance cost is:

balanceCost(CPU,RAM, 2) = max(0, 2 6 7) + max(0, 2 3 11)= 5 + 0

= 5.

If process p1 is moved from m1 to m2, then the cost is:

4

balanceCost(CPU,RAM, 2) = max(0, 2 8 15) + max(0, 2 1 3)= 1 + 0

= 1.

1.3.3 Process move cost

Some processes are painful to move; to model this soft constraint a processmove cost is defined. Let PMC(p) be the cost of moving the process p from itsoriginal machine M0(p).

processMoveCost =

pP such thatM(p) 6=Mo(p)

PMC(p)

For instanceM = {m1,m2} and P = {p1, p2}, M0(p1) = m1, M0(p2) = m1,PMC(p1) = 1, PMC(p2) = 10

5 and process p2 is moved from machine m1 tomachine m2, M(p2) = m2. Then processMoveCost = 10

5.

1.3.4 Service move cost

To balance moves among services, a service move cost is defined as the maximumnumber of moved processes over services. More formally:

serviceMoveCost = maxsS

({p s | M(p) 6= M0(p)})Consider for instanceM = {m1,m2,m3}, P = {p1, p2, p3, p4}, S = {{p1, p2}, {p3, p4}},

M0(p1) = m1, M0(p2) = m2, M0(p3) = m1, M0(p4) = m2, M(p1) = m2,M(p2) = m1, M(p3) = m1 and M(p4) = m3. Then serviceMoveCost =max(2, 1) = 2.

1.3.5 Machine move cost

Let MMC(msource,mdestination) be the cost of moving any process p from ma-chine msource to machine mdestination. Obviously for any machine m M,MMC(m,m) = 0. The machine move cost is then the sum of all moves weightedby relevant MMC:

machineMoveCost =pP

MMC(M0(p),M(p))

Consider for instance M = {m1,m2}, P = {p1, p2, p3, p4}, M0(p1) = m1,M0(p2) = m1, M0(p3) = m2, M0(p4) = m2, M(p1) = m2, M(p2) = m1,M(p3) = m1 and M(p4) = m1. MMC matrix is: MMC(m1,m1) = 0,MMC(m1,m2) = 10, MMC(m2,m1) = 7 and MMC(m2,m2) = 0. ThenmachineMoveCost = 10 + 0 + 7 + 7 = 24.

5

1.3.6 Total objective cost

The total objective cost is a weighted sum of all previous costs. It is the cost tominimize.

totalCost =rR

weightloadCost(r) loadCost(r)

+bB

weightbalanceCost(b) balanceCost(b)

+ weightprocessMoveCost processMoveCost+ weightserviceMoveCost serviceMoveCost+ weightmachineMoveCost machineMoveCost

2 I/O file formats

To ease the file format description, consider the following example with fourmachines, three processes, two resources and two services.

M = {m1,m2,m3,m4}P = {p1, p2, p3}R = {r1, r2}S = {sa, sb} with sa = {p1, p2}, sb = {p3}N = {n1, n2} with n1 = {m1,m2}, n2 = {m3,m4}L = {l1, l2, l3} with l1 = {m1,m2}, l2 = {m3}, l3 = {m4}T R = {r1}B = {r1, r2, 20}

C(m1, r1) = 30, C(m1, r2) = 400, SC(m1, r1) = 16, SC(m1, r2) = 80C(m2, r1) = 10, C(m2, r2) = 240, SC(m2, r1) = 8, SC(m2, r2) = 160C(m3, r1) = 15, C(m3, r2) = 100, SC(m3, r1) = 12, SC(m3, r2) = 80C(m4, r1) = 10, C(m4, r2) = 100, SC(m4, r1) = 8, SC(m4, r2) = 80R(p1, r1) = 12, R(p1, r2) = 10R(p2, r1) = 10, R(p2, r2) = 20R(p3, r1) = 6, R(p3, r2) = 200

sb depends on sa

spreadMin(sa) = 2, spreadMin(sb) = 1PMC(p1) = 1000, PMC(p2) = 100, PMC(p3) = 1

MMC(mi,mj) m1 m2 m3 m4m1 0 1 4 5m2 1 0 3 4m3 4 3 0 2m4 5 4 2 0

6

weightloadCost(r1) = 100, weightloadCost(r2) = 10weightbalanceCost(r1, r2, 20) = 10weightprocessMoveCost = 1weightserviceMoveCost = 10weightmachineMoveCost = 100

And the original solution is: M0(p1) = m1, M0(p2) = m4, M0(p3) = m1A new solution could be: M(p1) = m1, M(p2) = m3, M(p3) = m2

2.1 Instance input file format

In order to keep the file format as simple as possible, the instance input file isa list of integers and booleans. Values are space separated and should respectthe following order:

Number of resourcesFor each resource ri:

Boolean(is in T R)weightloadCost(ri)

Number of machinesFor each machine mi:

Neighborhood mi belongs toLocation mi belongs toCapacities, i.e. C(mi, r1) C(mi, r2) C(mi, r3). . .Safety capacties, i.e. SC(mi, r1) SC(mi, r2) SC(mi, r3). . .MMC(mi, ), i.e. MMC(mi,m1) MMC(mi,m2) MMC(mi,m3). . .

Number of servicesFor each service s:

spreadMin(s)Number of services s depends on and the list of those services

e.g. 3 sa sd se

Number of processesFor each process pi:

Service pi belongs toRequirements, i.e. R(pi, r1) R(pi, r2) R(pi, r3). . .PMC(pi)

Number of balance objectivesFor each balance objective bi:

balance triple, i.e. rj rk targetweightbalanceCost(bi)

weightprocessMoveCostweightserviceMoveCostweightmachineMoveCost

7

Next table illustrates the instance input format using previous instance asan example:

2 Number of resources

1 Resource #0 is transient100 weightloadCost of resource #0

0 Resource #1 is not transient10 weightloadCost of resource #1

4 Number of machines

0 Machine #0 is in neighborhood #00 Machine #0 is in location #030 400 Capacities of machine #016 80 Safety capacities of machine #00 1 4 5 Moving cost from machine #0 to machines #0, #1, #2 and #3

0 Machine #1 is in neighborhood #00 Machine #1 is in location #010 240 Capacities of machine #18 160 Safety capacities of machine #11 0 3 4 Moving cost from machine #1 to machines #0, #1, #2 and #3

1 Machine #2 is in neighborhood #11 Machine #2 is in location #115 100 Capacities of machine #212 80 Safety capacities of machine #24 3 0 2 Moving cost from machine #2 to machines #0, #1, #2 and #3

1 Machine #3 is in neighborhood #12 Machine #3 is in location #210 100 Capacities of machine #38 80 Safety capacities of machine #35 4 2 0 Moving cost from machine #3 to machines #0, #1, #2 and #3

2 Number of services

2 spreadMin of service #00 Service #0 doesnt depend on other services

1 spreadMin of service #11 Service #1 depends on one service0 Service #1 depends on service #0

8

3 Number of processes0 Process #0 is a process of service #012 10 Requirements of process #01000 Process Move Cost of process #0

0 Process #1 is a process of service #010 20 Requirements of process #1100 Process Move Cost of process #1

1 Process #2 is a process of service #16 200 Requirements of process #21 Process Move Cost of process #2

1 Number of balance costs0 1 20 Triple resource #0, resource #1, target 20 for balance cost #010 weight for balance cost #0

1 Weight of Process Move Cost10 Weight of Service Move Cost100 Weight of Machine Move Cost

2.2 Solution input/output file format

The same file format is used to define the original solution (input) and theoptimized solution (output). This file format is simply a list of assignment forall processes. As the number of processes is defined in the instance input file,machine indices are enough to define a solution.

Then the input file for the previous example is:

0 3 0

The total cost of this original solution is (400+1300)+2500+0+0+0 = 4200.Moving process p2 from machine m4 to m3 reduces the balance cost from

2500 to 1700 and the load cost of resource r1 from 400 to 200. The new totalcost is: (200 + 1300) + 1700 + 100 + 10 + 200 = 3510. Then optimal solutionis achieved by moving process p3 from machine m1 to m2 with a total cost of400 + 1600 + 101 + 10 + 300 = 2411.

The corresponding output file is:

0 2 1

9

2.3 Variable ranges for this challenge

The aim of this challenge is to concentrate on the optimization problem, there-fore set sizes are limited to:

Number of machines: 5,000

Number of resources: 20

Number of processes: 50,000

Number of services: 5,000

Number of neighborhoods: 1,000

Number of dependencies: 5,000

Number of locations: 1,000

Number of balance costs: 10

All other integers are indices or 32-bits unsigned integers.As usual in the ROADEF/EURO Challenge, three data sets will be provided:

Data set A: number of processes is limited to 1,000. This small data setis public and is used during the qualification phase;

Data set B: number of processes varies from 5,000 to 50,000. This medium/ large data set is public and is used to evaluate proposed solvers;

Data set X: number of processes varies from 5,000 to 50,000. This medium/ large data set is private and is used to evaluate proposed solvers.

2.4 Solution checker

In order to check if a produced solution is valid or not, and to compute the totalobjective cost, the source code of a solution checker is available. The syntax is:

solution checker instance filename original solution filename new solution filename

This solution checker will be used during the challenge to evaluate producedsolutions. Note the aim is to check the solution, not the instance; so the solutionchecker assumes the instance and the original solution are valid.

10