Research Article Genetic Scheduling and Reinforcement Learning …downloads.hindawi.com/journals/mpe/2015/597956.pdf · 2019-07-31 · Genetic Scheduling and Reinforcement Learning

Research ArticleGenetic Scheduling and Reinforcement Learning inMultirobot Systems for Intelligent Warehouses

Jiajia Dou12 Chunlin Chen12 and Pei Yang12

1Department of Control and Systems Engineering School ofManagement and Engineering Nanjing University Nanjing 210093 China2Research Center for Novel Technology of Intelligent Equipments Nanjing University Nanjing 210093 China

Correspondence should be addressed to Chunlin Chen clchennjueducn and Pei Yang yangpeinjueducn

Received 8 August 2015 Accepted 15 December 2015

Academic Editor Luciano Mescia

Copyright copy 2015 Jiajia Dou et alThis is an open access article distributed under theCreativeCommonsAttribution License whichpermits unrestricted use distribution and reproduction in any medium provided the original work is properly cited

A new hybrid solution is presented to improve the efficiency of intelligent warehouses with multirobot systems where the geneticalgorithm (GA) based task scheduling is combined with reinforcement learning (RL) based path planning for mobile robotsReinforcement learning is an effective approach to search for a collision-free path in unknown dynamic environments Geneticalgorithm is a simple but splendid evolutionary search method that provides very good solutions for task allocation In orderto achieve higher efficiency of the intelligent warehouse system we design a new solution by combining these two techniquesand provide an effective and alternative way compared with other state-of-the-art methods Simulation results demonstrate theeffectiveness of the proposed approach regarding the optimization of travel time and overall efficiency of the intelligent warehousesystem

1 Introduction

Intelligent warehouses are an essential part of the material-handling industry and have gained popularity in recentyears [1] Efficient and sensitive warehousing with intelligentmobile robots is critical to improve the overall productivityand simultaneously achieve high efficiency The operationswith respect to warehousing systems have received consider-able and increasing attention [2 3] In this paper we focus onthe path planning problem of autonomousmobile robots andthe scheduling issues of the task orders The path planningproblem [4] is to generate a collision-free path from thestarting point to the ending point and it is recognized as afundamental issue in warehousing systems Task scheduling[5] is the method to assign tasks to the robots under certaincriteria

The scheduling and path planning problems are a hottopic in multirobot systems for warehouses and many effec-tive methods have been developed over the past few decadesIn 1996 Jennings [6] introduced a task negotiation approachfor groups of autonomous robots in warehousing systemsHowever the coordination approach presented in that paper

cannot be applied to the uncertain environment The com-binatorial auction algorithm proposed by Sandholm [7] isconsidered to be more efficient than traditional mechanismsas it divides tasks into different clusters based on the correla-tions between them Then the robots bid on the unrestrictedcombinations of tasks A path planning approach combiningthe fuzzy method with the adaptive119876-leaning algorithm wasproposed in [8] and it was merely adapted to robot soccersystems In 2004 Richards et al [9] used a standard119860lowast searchalgorithm to find the optimal path Nevertheless the methodis only for the controlled and known environment

Most previous research of warehousing operations con-centrated on minimizing the travel distance where the work-load among the robots is not well balanced and the utilizationofmultirobot systems is not fully optimized Fewpublicationscould be found about the travel time In 2011 Elango et al [10]used the119870-means clustering and auction-based mechanismsto assign tasks to the robots in a balanced manner Thisapproach takes two objectives into consideration that istotal travel cost minimization and the workload balancingamong the robots However the complexity of the algorithmis comparatively high Zhou et al [11] proposed a balanced

Hindawi Publishing CorporationMathematical Problems in EngineeringVolume 2015 Article ID 597956 10 pageshttpdxdoiorg1011552015597956

2 Mathematical Problems in Engineering

heuristic mechanism to solve the task allocation problemin an intelligent warehouse which improves the balanceperformance among the robots and reduces the total traveltime effectively

Unfortunately even if there have been considerableresearches on scheduling and path planning problems com-binations of these two aspects have rarely been studied indetail In many cases these two issues are even considered tobe independentThe truth is that the assignment results fromthe path-length-based scheduling process and in turn affectsthe performance of the path planning approach Howeverthis relationship is often ignored in the previous researchesOur work considers the two issues simultaneously Weemploy reinforcement learning (RL) to explore the dynamicand unknown circumstance and use genetic algorithm (GA)to generate balanced and efficient task assignments for therobotsWe provide three criteria [11] to evaluate the proposedalgorithms that is travel time (TT) total travel cost (TTC)and balancing utilization (BU)We demonstrate the effective-ness of the proposed methods regarding the optimization oftravel time and overall efficiency for the warehousing systemthrough simulations

The rest of this paper is organized as follows In Section 2an introduction to the intelligent warehousing systems isgiven including the basic structures and the mathematicmodels In Section 3 we discuss the scheduling and pathplanning problems in warehousing systems Then the pro-posedmethods combiningGA andRL are described in detailIn Section 4 we elaborate on the implementations of thenew solution in an intelligent warehouse system as well ascomparisons with some other relevant algorithms Then theexperimental results are presented and analyzed Finally weconclude this paper in Section 5

2 Problem Statement

In this paper we focus on the task allocation and path plan-ning problems for the intelligent warehouses with multirobotsystems In a practical warehouse (eg the Kiva MobileFulfillment System [12 13]) there are generally dozens oreven hundreds ofmobile robots grabbing andmoving shelvesdirectly to workers at the picking station according to theactual task orders In order to simplify the system we makethe following assumptions about the robots tasks and theenvironment

(i) A grid-based map model is used for the warehouseenvironment

(ii) The environment of the system is unknown to therobots

(iii) The robot is equipped with necessary sensors that candetect the environment

(iv) The robot can get to an adjacent site through one ofthe four actions up down left and right

(v) Both the robots and tasks are identical with eachother

(vi) The robots are essentially independent when per-forming the tasks

Storage shelvesRobot

Picking station Pod

Figure 1 Simplified configuration of an intelligent warehousesystem

(vii) The total cost and the whole time to complete the taskare measured by the walking steps of the robots

(viii) The elapsed time at the station and lifting the shelf isfixed and constant

The simplifiedmodel of a specific intelligent warehouse isshown in Figure 1The system is composed of storage shelvesrobots and picking stations Each storage shelf containsseveral pods and each of them holds a certain number ofproducts The robot lifts and carries a corresponding pod tothe picking station along a learned path and then returns thepod to the original position based on the allocated orders It isclear that both task allocation and path planning algorithmsare crucial in getting the work done as soon as possible

Task allocation is themethod to assign the unfulfilled taskorders to the robots It is one of the most critical problemsin warehouse systems Path planning for a single robot canbe defined as searching for an optimal or suboptimal pathfrom the starting point to the destination regarding distanceand time which is considered to be a fundamental issue inwarehousing systems Furthermore these two problems areclosely correlated with each other and their relationship istaken into account in this paper We emphasize these twoissues simultaneously and attempt to accomplish the tasks asquickly as possible to improve the efficiency of warehousingsystems

In a typical warehousing system there always existnumerous unfulfilled task orders First we are confrontedwith a problem of how to assign the available tasks to alimited number of working robots By making wise decisionsabout task scheduling we can reduce the time spent to getthe work done Afterwards when the robots start to executetheir individual tasks we should consider the path planningproblem The goal is to find a collision-free path to thedestination and avoid obstacles with as few steps as possible

It is clear that at a high level our ultimate goal is to keepthe robots as busy as possible and maximize the efficiencyof the whole system Thus several relevant targets need tobe taken into account The first objective we considered isreducing the amount of time spent As the simulated runningtime is closely related to the performance of computersystems the running time is measured by the largest walkingsteps among the robots The second objective is to minimize

Mathematical Problems in Engineering 3

T1

T2

t11 t12

t14t13

t22

t23

t21

Assigned to r1

Assigned to r2

Task T1

Task T2

r1 r2

middot middot middot

Dr1t14

Dr2t22

c t 12t 13

c t 21t 23

ct14 t22

Figure 2 Basic parameters and indexes for an intelligent warehousewith multiple robots

the total distances of all the actual routes In addition inorder to balance the workload of the autonomous robots thebalancing condition is taken into consideration as well

We assume that there are a set of 119899 tasks119879 = 1199051 1199052 119905

119899

to be performed by a set of119898 robots 119877 = 1199031 1199032 119903

119898 The

travel cost between each pair of distinct tasks 119905119894 119905119895 (119905119894 119905119895isin

119879) is represented as 119888119894119895 Apparently the cost satisfies 119888

119894119895= 119888119895119894

in all cases The travel cost between a robot 119903119894and a task 119905

119895

is expressed as 119889119894119895 We define 119908

119894as the cost of task 119905

119894 which

is the total cost of carrying the inventory pod to the pickingstation according to task 119905

119894and then taking it backThe values

of 119888119894119895119889119894119895 and119908

119894are all calculated by path planning algorithms

which depend only on the actual path lengthsSuppose the tasks are divided into 119898 groups 119879 = 119879

1

1198792 119879

119898 and each section 119879

119894is allocated to robot 119903

119894 We

assume that 119903119894have 119896 tasks to perform 119879

1198941= 1199051198941 1199051198942 119905

119894119896

then the total cost of all the tasks in 119879119894can be expressed as

119882(119903119894) = 119908

1198941+ 1199081198942+ sdot sdot sdot + 119908

119894119896 The basic parameters and

indexes used in the model are introduced in detail as shownin Figure 2

We define the individual travel cost ITC119903119894 119879119894 which

indicates the total travel cost for robot 119903119894to complete its

assigned task set 119879119894and it can be computed by

ITC (119903119894 119879119894) = 119863

1199031198941199051198941+

119896minus1

sum

119895=1

119888119905119894119895119905119894(119895+1)

+119882(119903119894) (1)

where 1198631199031198941199051198941

represents the travel cost for robot 119903119894to reach its

first assigned task 1199051198941and sum119896minus1

119895=1119888119905119894119895119905119894(119895+1)

is the total travel costfor the other (119896 minus 1) tasks

In this paper three criteria are used to evaluate theperformance of different solutions travel time (TT) whichrepresents the maximum of all the individual travel costsamong the robots total travel cost (TTC) which indicates thetotal travel cost of all the robots in the system and balancingutilization (BU) which assesses the balancing condition of

R1 R2 Rm

Order input

Cost calculated

Task allocation

R

Path planning Path planning Path planning

Taskcompletion

Accurate

Centralcontroller

wij cij dij


Figure 3 Overall control structure

the whole system The corresponding formulas [11] are listedas follows

TT = max119894

ITC (119903119894 119879119894)

TTC =119898

sum

119894=1

ITC (119903119894 119879119894)

BU =min119894ITC (119903

119894 119879119894)

max119894ITC (119903

119894 119879119894)

(2)

3 GA and RL for Warehouse Scheduling andMultirobot Path Planning

31 Overall Control Structure In this paper the workspaceis divided into numerous grid cells with storage shelves inthe middle and picking stations around the perimeter Therobots are working on this rasterized map When an order isreceived the robot retrieves the appropriate pod and bringsit to the worker standing at inventory stations Then theappropriate items are picked packed and shipped out of thewarehousing system

Throughout the above process the task allocation processis executed by the central controller whereas the path plan-ning is implemented by the robots During the task allocationprocess the central controller receives a set number of taskorders from the Task Pool and employs GA [14ndash17] to assigntasks to the robots After that the central controller transmitsthe assignment results to the robots via wireless communica-tion and the robots execute the tasks progressively by using119876-learning [18ndash23] (a widely used RL algorithm) to plan pathsall along since the environment is unknown Figure 3 gives abrief overview of the whole process

Another point that must be noted is the collision avoid-ance between robots since the reinforcement learning algo-rithm in this paper is only for a single robot The interactionwhich happens when the path of one robot is blocked byanother robot or two or more robots attempt to move intothe same cell at a given time is not only variable but alsounpredictable A straightforward mechanism that may tacklethis problem is to refrain from these collisions When robots


(1) Initialize parameters population size Popsize maximal generations MaxEpoc crossover rate Pc(2) Calculate the distances between different task orders(3) Generate an population of feasible solutions randomly(4) Evaluate individuals in the initial population(5) for 119894 = 1 to MaxEpoc do(6) Set the alterable mutation rate Pm according to the actual evolution generations(7) Select individuals according to elitist strategy and roulette wheel method(8) if random (0 1) lt Pc then(9) Crossover individuals in pairs by a variation of the order crossover operator(10) end if(11) if random (0 1) lt Pm then(12) Mutate an individual by swap mutation(13) end if(14) Evaluate the produced offspring(15) end for

Algorithm 1 Genetic algorithm for warehouse scheduling

detect the others at short range (usually a few grids in frontof them) they enter into negotiation and in general casethe robot with fewer tasks left gives up the right-of-way Thismechanism is critical to the security of the whole system

32 GA Based on Virtual Task for Warehouse Scheduling GAis a search method inspired from natural evolution and hasbeen successfully applied to a variety of NP-hard combinato-rial optimization problems [14ndash17] It is iterative population-based algorithm and follows the principle of survival of thefittest to search solutions through the operations of selectioncrossover and mutation To apply GA to solve the warehousescheduling problem the focus is to establish an effective chro-mosome representation and design suitable selection cross-over and mutation operators to improve the solution quality

In this paper a concept of virtual task is introducedto code the multirobot task allocation problem meanwhileevaluation function and genetic operators are also specificallydesigned for the issue An improved GA for warehousescheduling is presented in Algorithm 1 The entire algorithmterminates when a fixed evolution generation is reached

321 Chromosome Representation Chromosome represen-tation is particularly important in GA and a specific problemmust be indicated by the appropriate code to promote theevolutionary process In this paper we use a single integerstring of length (119898+119899minus1) as representation of chromosomewhere119898 is the number of robots and 119899 is the number of tasksIn the chromosome 119899 tasks are represented by a permutationof integers from 1 to 119899The string is portioned into119898 sectionsby adding (119898minus 1) virtual tasks that is (119899 + 1) (119899 + 2) sdot sdot sdot (119899 +119898 minus 1) which indicate the transition from one robot to theother

An example of task allocation for 8 tasks with 3 robots isgiven to demonstrate the above chromosome representationmeasure We only need to add 2 virtual tasks in the chromo-some that is 9 and 10 to be the division pointsThe sequencesin Table 1 are the possible situations for the task allocation

During the process of evolution the virtual tasks in thechromosome may appear at both ends or be adjacent to each

Table 1 Task allocation coding and the corresponding distributionresults

Chromosomes Distribution results3 5 6 9 2 7 10 1 4 8 R1 3 5 6 R2 2 7 R3 1 4 89 3 4 8 10 2 1 6 5 7 R1 0 R2 3 4 8 R3 2 1 6 5 72 5 7 10 9 1 8 4 6 3 R1 2 5 7 R2 0 R3 1 8 4 6 36 3 1 4 9 2 5 8 7 10 R1 6 3 1 4 R2 2 5 8 7 R3 0

other Hence there will be at least one robot who has no tasksto perform which will lead to unbalance in the warehousethus this kind of chromosome will be eliminated during theevolution

322 Design of Fitness Function Fitness functions are usedin GA to evaluate the quality of the candidate solutionsWhen the initial population is generated or an offspringis produced the fitness values of the feasible solutions arecalculated by a fitness functionThehigher the fitness value isthe better the individual is In other words a fitness functioncan facilitate the evolution event by distinguishing the goodsolutions from the bad ones However there is no suchbenchmark for a fixed fitness function We need to designan appropriate fitness function based on the characteristicsof our warehousing systems

In the case of warehouse scheduling we aim to achievethree different objectives (1) minimizing the overall time toperform the tasks TT (2) minimizing the total travel costof all the robots TTC (3) and maximizing the balancingcondition of task allocation BU Hence the fitness functionincludes three different parts accordingly For the first twoparts the reciprocals of TT and TTC are used to be the eval-uation criteria thus the minimum solutions are convertedto the maximum ones Taking MaxTest and TotalTest as thetwo constants related to the actual task numbers the fitnessfunction can be defined as

119865 (119909) =MaxTest

TT+TotalTestTTC

+ BU (3)


323 The Genetic Operators

(a) Selection Operator Selection selects a certain number ofindividuals based on the fitness value In general the wholeprocess follows the principle of survival of the fittest Inother words the individuals with higher fitness values aremore likely to be selected to the next generation than theothers In this paper we combine elitist strategy with roulettewheel method as the selection strategy First we sort theindividuals at present according to their own fitness valueand copy the top 20 directly to the next generation Thiselitist strategy can prevent the outstanding individuals frombeing eliminated after a selection operator which is a vitalguarantee for the convergence of the algorithmThen roulettewheel method is used to choose the remaining individualsRoulette wheel method is also named as proportional modeland the possibility of being selected is proportional to thefitness of each individual The better the individuals are themore chances to be inherited to the next generation they haveThis new selection strategy is proved to be valid and efficientduring the evaluation

(b) Crossover Operator After selection individuals arecrossed over to create various and promising offspring Itis commonly recognized that crossover operation is the keymethod to generate new individuals Furthermore it is animportant feature which distinguishes GA from other evo-lutionary algorithms Similarly the top 20 individuals aredirectly inherited to the next generation For the rest a vari-ation of the order crossover (OX) [14] operator is introduced

Our crossover operator is responsible for the recom-bination of two individuals to form two new offspringThus the new individuals will inherit some of the featuresfrom the first parent and retain some characteristics of theother Specifically the new individuals are constructed in thefollowing way Regardless of those outstanding individualswho will not be crossed over the remainders are paired as theparent individuals in accordance with the sequences in thepopulation After that for each pair two crossing points needto be determined Unlike the traditional methods we selectthe first crossing point in the anterior half of the individualand the second in the latter half This measure has wellcontrolled the length of the subsection determined by thetwo cut points which will influence the evolution rate andthe convergence of the algorithm Finally we construct newindividuals by coping with the subsection of one parent andretaining the relative order of genes of the other one

For example if there are two parent individuals

(9 3 4 8 10 2 1 6 5 7)

(2 5 7 10 9 1 8 4 6 3) (4)

suppose that the substrings between the vertical lines areselected

(9 3 4 | 8 10 2 1 | 6 5 7)

(2 5 7 | 10 9 1 8 | 4 6 3) (5)

(9 3 4 8 10 2 1 6 5 7) (2 5 7 10 9 1 8 4 6 3)

(8 10 2 1 5 7 9 4 6 3)(10 9 1 8 3 4 2 6 5 7) Off2Off1

Figure 4 The crossover procedure

The corresponding segments are directly duplicated into theoffspring respectively which result in

(8 10 2 1 lowast lowast lowast lowast lowast lowast )

(10 9 1 8 lowast lowast lowast lowast lowast lowast ) (6)

Next copy the genes that are not presented in theoffspring yet from the other parent in the order they appearin In this example two new offspring are produced

(8 10 2 1 5 7 9 4 6 3)

(10 9 1 8 3 4 2 6 5 7) (7)

The procedure of the above operators is described in Figure 4

(c) Mutation Operator Mutation takes charge of addingnew traits to the individuals thus maintaining variability ofthe population It is indispensable in preventing prematuretermination in GA by improving the local search abilityIt should be noted that in order to preserve the currentexcellent individuals the top 20 do not participate in themutation operation We adopt swap mutation (SM) [14]operator in this task scheduling algorithm for the otherindividualsThe swapmutation operator which is also namedas the point mutation operator randomly selects two genes inthe chromosome and exchanges them For example considerthe following sequence

(5 3 6 9 2 7 10 1 4 8) (8)

and assume that the second and the fifth genes are randomlyselected we swap the two points simultaneously and find thatit turns into

(5 2 6 9 3 7 10 1 4 8) (9)

In this operation we swap the characteristics of individualsgoverned by the mutation rate at a certain extent Anappropriate mutation rate may ensure the ability to explorethe entire search space without weakening the performanceof the algorithm Thus we select different values accordingto different requirements Initially the mutation operation isperformed with a low probability of 01 to preserve the vitalindividuals After 1000 generations the evolution processtends to be stable and we reset the mutation rate at a highervalue of 05 to search for the new individuals

33 119876-Learning for Path Planning of Multirobot SystemsReinforcement learning (RL) [23ndash26] is a widely used learn-ing method in cybernetics statistics mental philosophy andso forth It has been an active branch in AI for decades RLis a powerful mechanism that does not require any prior


(1) Initialize the total episodes of learning Maxepi maximum step number Maxstep discount rate 120574(2) Initialize the coordinate of the starting and ending point(3) Initialize 119876(119904 119886) based on the actual grid world(4) for 119894 = 1 to Maxepi do(5) Initialize the learning rate 120572(6) for 119895 = 1 to Maxstep do(7) Decrease the learning rate 120572 gradually(8) Choose 119886 from 119904 by greedy action selection(9) Take action 119886

119905 observe 119903

119905+1 119904119905+1

(10) 119876(119904

119905 119886119905) larr 119876(119904

119905 119886119905) + 120572[119903

119905+1+ 120574max

119886isin119860119876(119904119905+1 119886) minus 119876(119904

119905 119886119905)]

(11) 119904 larr 119904119905+1

(12) if 119904119905+1

is a goal state then(13) Break(14) end if(15) end for(16) end for

Algorithm 2 119876-learning for path planning

knowledge and the robot can autonomously learn to improveits performance over time by the trial-and-error interactingwith its working environment The only response from theenvironment is a delayed reward and the goal of the robot istomaximize the accumulated rewards in the long run for eachaction In this kind of learning scheme the system designersjust need to provide the ultimate goal to be achieved insteadof telling the robot how to reach the target On accountof the above characteristics RL has become a hotspot ofresearch and is a promisingmethod to solve the path planningproblem for mobile robots

The particular RL algorithm that we employ for theproblem of path planning here is119876-learning119876-learning [18ndash23] is a kind of model-free RL algorithms which has beenwidely used due to its simplicity and theory maturity 119876-learning uses the action-value function119876(119904 119886) to indicate theestimated values for each state-action pair and then recur-sively finds the optimal solution The procedural form of 119876-leaning for path planning is presented inAlgorithm 2 Severalmajor issues about Algorithm 2 are addressed as follows

331 The Reward Function The robots proceed their learn-ing course through the rewards they obtained after a certainaction is taken The reward value is a feedback from theenvironment which evaluates the action in an immediatesense A reward function indicates how good the robotrsquosperformance is and thus defines the goal in RL problem Inthis paper a frequently employed reward strategy is adoptedIn the grid world the purpose of the robot is to find anoptimal collision-free path from the starting point to theending point Therefore upon reaching the correspondinggoal the robot will attract a large positive reward of 200 Anegative reward of minus50 is received as a punishment when therobot collides into a wall In other cases a reward of minus1 isprovided so as to encourage fewer steps In this way the robotwill prefer those actions which produce higher rewards overthe long run in the process of learning In other words thisreward function will prompt the robot to follow the optimalpath that results in the best possible accumulated rewards

332 The Value Function Unlike the reward function thevalue function picks out those actions which obtain for usthe maximal rewards over the long run The target of thelearning robot is to maximize the accumulated rewards peraction it receives from the environment 119876-learning findsan optimal strategy by learning the values of the so-called119876 function The function 119876(119904 119886) is defined as the expecteddiscounted cumulative reward that is received by the robotwhen executing action 119886 in state 119904 and performing an optimalstrategy afterwards

We start with an initial estimation of 119876(119904 119886) for eachstate-action pair based on the actual grid world and the valueis updated by interacting with the dynamic environmentcontinuously In the course of approaching the goal the nextstate of the environment is determined by the current stateand the action the robot takes At a certain step 119905 the robotacquires the current state of the environment 119904

119905isin 119878 where 119878 is

the set of all possible states and then takes an action 119886119905isin 119860

where 119860 is the set of actions available in state 119904119905 After the

action 119886 is taken an immediate reward 119903119905+1

of the state-actionpair is sent to the robot and the environment is in a new state119904119905+1

The corresponding updated rule [23] is as follows

119876 (119904119905 119886119905)

larr997888 119876 (119904119905 119886119905)

+ 120572 [119903119905+1+ 120574max119886isin119860

119876 (119904119905+1 119886) minus 119876 (119904

119905 119886119905)]

(10)

where max119886isin119860119876(119904119905+1 119886) is the maximum 119876-value calculated

for taking all possible actions on the next state 119904119905+1

Theparameter 120574 isin [0 1] represents the discount factor thatcontrols howmuch impact future rewards have on the presentdecision-making With a small value of 120574 the robot tends toattach more importance to the near-term rewards whereasthe robot will take the future rewards into consideration as 120574becomes closer to 1 The parameter 120572 isin [0 1] is the learn-ing rate A small value of this parameter means the learningprocess will proceed at a slow speed while a too high value


might affect the convergence property of the algorithmObviously an appropriate learning rate is critical to theperformance of the whole algorithm In this paper thelearning rate is alterable and decreased step by step during theprocess of learning Finally it will approach a relative smallvalue The action-value function 119876(119904 119886) is updated at eachtime step of an episode until the robot enters into the goalstate It has been proved that if all state-action pairs are visitedinfinitely often by the robot and an appropriate learning rate isselected the estimated 119876(119904 119886) will converge with probabilityone to the optimum value 119876lowast(119904 119886)

333 Action Selection Strategies In the physical world arobot usually can move in all directions Here the actionspace is simplified by assuming that only five basic move-ments are available for each robot up down left right andstand At each time step the robot can select any of them aslong as there is an admissible transition When a robot triesto implement an action that would take it out of bounds themotion will not happen

In the path planning problem in order to find a collision-free path within the least amount of steps the robot needs toemploy asmany optimal action selection strategies as possiblein the process of learning The task of a robot is thus to learna policy 120587 that maps the current state 119904 into the desirableaction 119886 to be performedThe action policy should be learnedthrough trial-and-error and through evaluating every state-action pair so that we can build a policy for working inthe intelligent warehouse During the process of learningif the action is selected according to the random principlesome unnecessary pairs which have little 119876-value will occurfrequently This can bring long learning time and the 119876-learning algorithm may not converge In this paper we usea deterministic selection mechanism called greedy policy tochoose actions A greedy strategy [23] refers to a policy thatthe robot invariably sorts out the action with the highestestimated 119876-value

120587lowast

(119904) = argmax119886

119876 (119904 119886) (11)

It assigns probability one to action argmax119886119876(119904 119886) in state

119904 and the selected action is called a greedy action Theexperiments show that the action selection strategy is quiteeffective and the learning speed is increased evidently

4 Experiments

In this section we test the performance of the proposedintegrated approach by applying it to a simulated intelligentwarehouse system As shown in Figure 5 the map of thewarehouse is divided into a 50 times 28 grid world where eachrobot and pod occupies a single grid exactly There are 4 times12 shelves in total located in the center and each storageshelf consists of 6 inventory pods Three picking stations aresituated in the bottom side of the whole system We have 8robots available working for the task orders and each timethey can move just one cell within the scope of the map

A Task Pool is used to store the dynamic entry-ordersand it will send a certain number of orders to the central


Picking station Pod

Figure 5 Rasterized map of the simulated intelligent warehousesystem

controller at a specific time After the controller receives thetasks from the Task Pool we apply various methods to assignthem to the robots Then the central controller sends theassignment results to the robots via wireless communicationand the robots start to work on their own tasks by using pathplanning algorithms to calculate the path all the way

41 Parameter Settings The parameter settings for GA areselected as follows the population size is set to 100 mutationprobability is primarily assigned to 01 and then transformedinto 05 after 1000 generations the crossover rate is set to095 which is usually kept high to increase the global searchability the total number of generations is assigned to be alarge integer 100000 to get a satisfactory result Parametersettings for 119876-learning are very critical since they have adirect impact on the rate of convergence and quality of thefinal results The discount factor is set to a constant number08 while the learning rate is typically time varying Theinitial value of learning rate is assigned to 03 and it decays byeach step during the leaning process Finally the learning ratewill reach a small value at the end of runThe maximum stepnumber for each episode is set as 15000 When the numberof the accumulated steps reaches 15000 it means failure toreach the target within the limited number of steps and theepisodewill be terminated and continued to the next oneThemaximum episodes for 119876-learning are set as 8000 to obtainreliable experimental results

42 Experimental Results In order to verify the effectivenessof the proposed approach we carried out several groups ofexperiments by combining different task allocation methods(ie GA auction-based method [7 11] 119870-means clusteringmethod [10] and random allocation) with different pathplanning methods (ie 119876-learning and classic heuristicalgorithm of 119860lowast [9 11]) which leads to eight groups ofsimulation experiments in thewarehousing system as follows

(1) GA with 119876-learning(2) GA with 119860lowast(3) Auction with 119876-learning(4) Auction with 119860lowast


50 100 150 200 250 300 350 4000

1000

2000

3000

4000

5000

6000

7000

8000

9000

Number of tasks

TT

GA + Q-learningGA + Alowast

Auction + Q-learningAuction + Alowast

K-means + Q-learningK-means + Alowast

Rand + Alowast

Rand + Q-learning

Figure 6 Travel time (TT) spent by each different approach

Number of tasks50 100 150 200 250 300 350 400

010000200003000040000500003000035000400004500050000

TTC




Rand + Q-learningRand + Alowast

Figure 7 Total travel cost (TTC) of each different approach

(5) 119870-means with 119876-learning(6) 119870-means with 119860lowast(7) Rand with 119876-learning(8) Rand with 119860lowast

The results of all these solutions are shown in Figures 6ndash8 where the performances are given regarding the criteria oftravel time TT total travel cost TTC and balancing utiliza-tion BU All the results are averaged over 20 simulations inthe above experimental environment

Figure 6 shows the time spent by the above eight groupsof experiments as the number of tasks increases It can beseen that GA performs much better than the others wherethe robots are able to accomplish the allocation task in amuch shorter time while the performance of119870-means is the

50 100 150 200 250 300 350 4000

102030405060708090

100

BU (

)

Number of tasks




Rand + AlowastRand + Q-learning

Figure 8 Balancing utilization (BU) of each different approach

worst This is because in the traditional 119870-means methodthe task allocation process concentrates more on minimizingthe travel cost and cannot control the balancing conditioneffectively Once there is a robot nearest to an open task otherrobots will not attempt to compete even if they are sittingaround idle and thus caused a bad behavior

Figure 7 displays the curves for the total travel cost It isclear that there is only a slight distinction between these eightgroups regardless of the number of tasks Unsurprisingly thetravel costs of 119870-means are the lowest among all the othersAnother important aspect is that119876-learning exhibits a similarperformance with 119860lowast In other words although 119876-learningdoes not use the model of the environment it can find theoptimal path as 119860lowast does

Figure 8 shows the balancing utilization and the perfor-mance of GA combined with 119860lowast or 119876-leaning are strikinglydifferent from the other solutions They have a steady valueof about 95 all along By contrast119870-means combined with119860lowast or119876-leaning perform significantly worseThe BU of them

is much smaller none of them reached 50 which indicatesa severe bias towards using of robots and poor balancingperformance In that condition the utilization of the system isquite unsatisfiedThe other four groups outperform119870-meansin most cases the BU values range from 25 to 75 as thetask number increases and the four methods perform veryclose to each other

5 Conclusion

In this paper we focus on the typical scheduling and pathplanning problems in an intelligent warehouse GA and 119876-learning are presented for task allocation and path planningrespectively In comparison we review and evaluate somepopular algorithms in these two issues for autonomousmobile robots in warehousing systems The experimentalresults demonstrate the effectiveness of the new solutionby combining GA and RL GA generates the best results


for warehouse scheduling compared to other methods weexamined The performance of 119860lowast is roughly the same with119876-learning for path planning problem However 119860lowast is onlysuitable for the case when the working environment is knownto the robots As RL does not necessarily need the model ofthe environment the robot gains the ability to adapt to differ-ent circumstanceThe new solution improves the utility of theintelligent warehouse system remarkably and is practical forvarious applications Our further work will focus on avoid-ing dynamic obstacles in a warehouse and transfer learn-ing for multiple robots [27ndash29]

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgment

This work was supported by the National Natural ScienceFoundation of China (nos 61273327 and 61432008)

References

[1] R de Koster T Le-Duc and K J Roodbergen ldquoDesign andcontrol of warehouse order picking a literature reviewrdquo Euro-pean Journal of Operational Research vol 182 no 2 pp 481ndash5012007

[2] P R Wurman R DrsquoAndrea and M Mountz ldquoCoordinatinghundreds of cooperative autonomous vehicles in warehousesrdquoAI Magazine vol 29 no 1 pp 9ndash19 2008

[3] J Gu M Goetschalckx and L F McGinnis ldquoResearch on ware-house operation a comprehensive reviewrdquo European Journal ofOperational Research vol 177 no 1 pp 1ndash21 2007

[4] T-K Wang Q Dang and P-Y Pan ldquoPath planning approachin unknown environmentrdquo International Journal of Automationand Computing vol 7 no 3 pp 310ndash316 2010

[5] B P Gerkey andM JMataric ldquoA formal analysis and taxonomyof task allocation in multi-robot systemsrdquo The InternationalJournal of Robotics Research vol 23 no 9 pp 939ndash954 2004

[6] N R Jennings Coordination Techniques for Distributed Artifi-cial Intelligence Wiley-Interscience 1996

[7] T Sandholm ldquoAlgorithm for optimal winner determination incombinatorial auctionsrdquo Artificial Intelligence vol 135 no 1-2pp 1ndash54 2002

[8] K-S Hwang S-W Tan and C-C Chen ldquoCooperative strategybased on adaptive Q-learning for robot soccer systemsrdquo IEEETransactions on Fuzzy Systems vol 12 no 4 pp 569ndash576 2004

[9] N D Richards M Sharma andD GWard ldquoA hybrid Alowastauto-maton approach to on-line path planning with obstacle avoid-ancerdquo in Proceedings of the 1st AIAA Intelligent Systems TechnicalConference pp 141ndash157 Chicago Ill USA September 2004

[10] M Elango S Nachiappan and M K Tiwari ldquoBalancing taskallocation inmulti-robot systems usingK-means clustering andauction based mechanismsrdquo Expert Systems with Applicationsvol 38 no 6 pp 6486ndash6491 2011

[11] L Zhou Y Shi J Wang and P Yang ldquoA balanced heuristicmechanism for multirobot task allocation of intelligent ware-housesrdquoMathematical Problems in Engineering vol 2014 Arti-cle ID 380480 10 pages 2014

[12] R DrsquoAndrea and PWurman ldquoFuture challenges of coordinatinghundreds of autonomous vehicles in distribution facilitiesrdquo inProceedings of the IEEE International Conference on Technolo-gies for Practical Robot Applications (TePRA rsquo08) pp 80ndash83Woburn Mass USA November 2008

[13] J Enright and R Wurman ldquoOptimization and coordinatedautonomy in mobile fulfillment systemsrdquo in Proceedings of theWorkshops at the 25thAAAIConference onArtificial Intelligencepp 33ndash38 San Francisco Calif USA August 2011

[14] P Larranaga C M H Kuijpers R H Murga I Inza andS Dizdarevic ldquoGenetic algorithms for the travelling salesmanproblem a review of representations and operatorsrdquo ArtificialIntelligence Review vol 13 no 2 pp 129ndash170 1999

[15] M F Tasgetiren P N Suganthan Q-K Pan and Y-C LiangldquoA genetic algorithm for the generalized traveling salesmanproblemrdquo in Proceedings of the IEEE Congress on EvolutionaryComputation (CEC rsquo07) pp 2382ndash2389 Singapore September2007

[16] J Li Q Sun M C Zhou and X Dai ldquoA new multiple travel-ing salesmanproblemand its genetic algorithm-based solutionrdquoin Proceedings of the IEEE International Conference on SystemsMan and Cybernetics (SMC rsquo13) pp 627ndash632 IEEE Manch-ester UK October 2013

[17] A Kiraly and J Abonyi ldquoOptimization of multiple travelingsalesmen problem by a novel representation based geneticalgorithmrdquo in Intelligent Computational Optimization in Engi-neering vol 366 of Studies in Computational Intelligence pp241ndash269 Springer Berlin Germany 2011

[18] C Chen D Dong H X Li and T J Tarn ldquoHybrid MDP basedintegrated hierarchical q-learningrdquo Science in China Series FmdashInformation Sciences vol 54 no 11 pp 2279ndash2294 2011

[19] A Konar I G Chakraborty S J Singh L C Jain and A KNagar ldquoA deterministic improved Q-learning for path planningof a mobile robotrdquo IEEE Transactions on Systems Man andCybernetics Systems vol 43 no 5 pp 1141ndash1153 2013

[20] C Chen H-X Li and D Dong ldquoHybrid control for robotnavigationmdasha hierarchical Q-learning algorithmrdquo IEEE Robo-tics and Automation Magazine vol 15 no 2 pp 37ndash47 2008

[21] C Chen D Dong H-X Li J Chu and T-J Tarn ldquoFidelity-based probabilistic Q-learning for control of quantum systemsrdquoIEEE Transactions on Neural Networks and Learning Systemsvol 25 no 5 pp 920ndash933 2014

[22] D Dong C Chen J Chu and T-J Tarn ldquoRobust quantum-inspired reinforcement learning for robot navigationrdquo IEEEASME Transactions on Mechatronics vol 17 no 1 pp 86ndash972012

[23] R S Sutton and A G Barto Reinforcement Learning An Intro-duction MIT Press Cambridge Mass USA 1998

[24] T Kollar and N Roy ldquoUsing reinforcement learning to improveexploration trajectories for error minimizationrdquo in Proceedingsof the IEEE International Conference on Robotics and Automa-tion (ICRA rsquo06) pp 3338ndash3343 Orlando Fla USA May 2006

[25] HHViet PHKyaw andTCChung ldquoSimulation-based eval-uations of reinforcement learning algorithms for autonomousmobile robot path planningrdquo in IT Convergence and Servicesvol 107 of Lecture Notes in Electrical Engineering pp 467ndash476Springer Amsterdam The Netherlands 2011

[26] M A K Jaradat M Al-Rousan and L Quadan ldquoReinforce-ment based mobile robot navigation in dynamic environmentrdquoRobotics and Computer-Integrated Manufacturing vol 27 no 1pp 135ndash149 2011


[27] M Tan ldquoMulti-agent reinforcement learning independent vscooperative agentsrdquo inProceedings of the 10th International Con-ference on Machine Learning (ICML rsquo93) pp 330ndash337 AmherstMass USA June 1993

[28] L Busoniu R Babuska and B De Schutter ldquoA comprehensivesurvey of multiagent reinforcement learningrdquo IEEE Transac-tions on Systems Man and Cybernetics Part C Applications andReviews vol 38 no 2 pp 156ndash172 2008

[29] Y Hu Y Gao and B An ldquoMultiagent reinforcement learningwith unshared value functionsrdquo IEEE Transactions on Cybernet-ics vol 45 no 4 pp 647ndash662 2015

Submit your manuscripts athttpwwwhindawicom

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

MathematicsJournal of


Mathematical Problems in Engineering

Hindawi Publishing Corporationhttpwwwhindawicom

Differential EquationsInternational Journal of

Volume 2014

Applied MathematicsJournal of


Probability and StatisticsHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Journal of


Mathematical PhysicsAdvances in

Complex AnalysisJournal of


OptimizationJournal of


CombinatoricsHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of


Operations ResearchAdvances in

Journal of


Function Spaces

Abstract and Applied AnalysisHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of Mathematics and Mathematical Sciences


The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014


Algebra

Discrete Dynamics in Nature and Society



Decision SciencesAdvances in

Discrete MathematicsJournal of


Volume 2014 Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Stochastic AnalysisInternational Journal of


heuristic mechanism to solve the task allocation problemin an intelligent warehouse which improves the balanceperformance among the robots and reduces the total traveltime effectively

Unfortunately even if there have been considerableresearches on scheduling and path planning problems com-binations of these two aspects have rarely been studied indetail In many cases these two issues are even considered tobe independentThe truth is that the assignment results fromthe path-length-based scheduling process and in turn affectsthe performance of the path planning approach Howeverthis relationship is often ignored in the previous researchesOur work considers the two issues simultaneously Weemploy reinforcement learning (RL) to explore the dynamicand unknown circumstance and use genetic algorithm (GA)to generate balanced and efficient task assignments for therobotsWe provide three criteria [11] to evaluate the proposedalgorithms that is travel time (TT) total travel cost (TTC)and balancing utilization (BU)We demonstrate the effective-ness of the proposed methods regarding the optimization oftravel time and overall efficiency for the warehousing systemthrough simulations

The rest of this paper is organized as follows In Section 2an introduction to the intelligent warehousing systems isgiven including the basic structures and the mathematicmodels In Section 3 we discuss the scheduling and pathplanning problems in warehousing systems Then the pro-posedmethods combiningGA andRL are described in detailIn Section 4 we elaborate on the implementations of thenew solution in an intelligent warehouse system as well ascomparisons with some other relevant algorithms Then theexperimental results are presented and analyzed Finally weconclude this paper in Section 5

2 Problem Statement

In this paper we focus on the task allocation and path plan-ning problems for the intelligent warehouses with multirobotsystems In a practical warehouse (eg the Kiva MobileFulfillment System [12 13]) there are generally dozens oreven hundreds ofmobile robots grabbing andmoving shelvesdirectly to workers at the picking station according to theactual task orders In order to simplify the system we makethe following assumptions about the robots tasks and theenvironment

(i) A grid-based map model is used for the warehouseenvironment

(ii) The environment of the system is unknown to therobots

(iii) The robot is equipped with necessary sensors that candetect the environment

(iv) The robot can get to an adjacent site through one ofthe four actions up down left and right

(v) Both the robots and tasks are identical with eachother

(vi) The robots are essentially independent when per-forming the tasks


Picking station Pod

Figure 1 Simplified configuration of an intelligent warehousesystem

(vii) The total cost and the whole time to complete the taskare measured by the walking steps of the robots

(viii) The elapsed time at the station and lifting the shelf isfixed and constant

The simplifiedmodel of a specific intelligent warehouse isshown in Figure 1The system is composed of storage shelvesrobots and picking stations Each storage shelf containsseveral pods and each of them holds a certain number ofproducts The robot lifts and carries a corresponding pod tothe picking station along a learned path and then returns thepod to the original position based on the allocated orders It isclear that both task allocation and path planning algorithmsare crucial in getting the work done as soon as possible

Task allocation is themethod to assign the unfulfilled taskorders to the robots It is one of the most critical problemsin warehouse systems Path planning for a single robot canbe defined as searching for an optimal or suboptimal pathfrom the starting point to the destination regarding distanceand time which is considered to be a fundamental issue inwarehousing systems Furthermore these two problems areclosely correlated with each other and their relationship istaken into account in this paper We emphasize these twoissues simultaneously and attempt to accomplish the tasks asquickly as possible to improve the efficiency of warehousingsystems

In a typical warehousing system there always existnumerous unfulfilled task orders First we are confrontedwith a problem of how to assign the available tasks to alimited number of working robots By making wise decisionsabout task scheduling we can reduce the time spent to getthe work done Afterwards when the robots start to executetheir individual tasks we should consider the path planningproblem The goal is to find a collision-free path to thedestination and avoid obstacles with as few steps as possible

It is clear that at a high level our ultimate goal is to keepthe robots as busy as possible and maximize the efficiencyof the whole system Thus several relevant targets need tobe taken into account The first objective we considered isreducing the amount of time spent As the simulated runningtime is closely related to the performance of computersystems the running time is measured by the largest walkingsteps among the robots The second objective is to minimize


T1

T2

t11 t12

t14t13

t22

t23

t21

Assigned to r1

Assigned to r2

Task T1

Task T2

r1 r2


Dr1t14

Dr2t22

c t 12t 13

c t 21t 23

ct14 t22




119899


119898 The



119894119895= 119888119895119894


119895



119894 which



of 119888119894119895119889119894119895 and119908



1

1198792 119879



119894 We


1198941= 1199051198941 1199051198942 119905

119894119896


119882(119903119894) = 119908

1198941+ 1199081198942+ sdot sdot sdot + 119908






ITC (119903119894 119879119894) = 119863

1199031198941199051198941+

119896minus1

sum

119895=1

119888119905119894119895119905119894(119895+1)

+119882(119903119894) (1)

where 1198631199031198941199051198941



119895=1119888119905119894119895119905119894(119895+1)



R1 R2 Rm

Order input

Cost calculated

Task allocation

R


Taskcompletion

Accurate

Centralcontroller

wij cij dij




TT = max119894

ITC (119903119894 119879119894)

TTC =119898

sum

119894=1

ITC (119903119894 119879119894)

BU =min119894ITC (119903

119894 119879119894)

max119894ITC (119903

119894 119879119894)

(2)



















119865 (119909) =MaxTest

TT+TotalTestTTC

+ BU (3)







(9 3 4 8 10 2 1 6 5 7)

(2 5 7 10 9 1 8 4 6 3) (4)


(9 3 4 | 8 10 2 1 | 6 5 7)

(2 5 7 | 10 9 1 8 | 4 6 3) (5)

(9 3 4 8 10 2 1 6 5 7) (2 5 7 10 9 1 8 4 6 3)

(8 10 2 1 5 7 9 4 6 3)(10 9 1 8 3 4 2 6 5 7) Off2Off1






(8 10 2 1 5 7 9 4 6 3)

(10 9 1 8 3 4 2 6 5 7) (7)



(5 3 6 9 2 7 10 1 4 8) (8)


(5 2 6 9 3 7 10 1 4 8) (9)





119905 observe 119903

119905+1 119904119905+1

(10) 119876(119904

119905 119886119905) larr 119876(119904

119905 119886119905) + 120572[119903

119905+1+ 120574max

119886isin119860119876(119904119905+1 119886) minus 119876(119904

119905 119886119905)]

(11) 119904 larr 119904119905+1

(12) if 119904119905+1








119905isin 119878 where 119878 is






119876 (119904119905 119886119905)

larr997888 119876 (119904119905 119886119905)

+ 120572 [119903119905+1+ 120574max119886isin119860

119876 (119904119905+1 119886) minus 119876 (119904

119905 119886119905)]

(10)








120587lowast

(119904) = argmax119886

119876 (119904 119886) (11)



4 Experiments




Picking station Pod







50 100 150 200 250 300 350 4000

1000

2000

3000

4000

5000

6000

7000

8000

9000

Number of tasks

TT




Rand + Alowast

Rand + Q-learning


Number of tasks50 100 150 200 250 300 350 400

010000200003000040000500003000035000400004500050000

TTC









50 100 150 200 250 300 350 4000

102030405060708090

100

BU (

)

Number of tasks










5 Conclusion






Acknowledgment


References






































Volume 2014




Journal of











Journal of


Function Spaces






Algebra










T1

T2

t11 t12

t14t13

t22

t23

t21

Assigned to r1

Assigned to r2

Task T1

Task T2

r1 r2


Dr1t14

Dr2t22

c t 12t 13

c t 21t 23

ct14 t22




119899


119898 The



119894119895= 119888119895119894


119895



119894 which



of 119888119894119895119889119894119895 and119908



1

1198792 119879



119894 We


1198941= 1199051198941 1199051198942 119905

119894119896


119882(119903119894) = 119908

1198941+ 1199081198942+ sdot sdot sdot + 119908






ITC (119903119894 119879119894) = 119863

1199031198941199051198941+

119896minus1

sum

119895=1

119888119905119894119895119905119894(119895+1)

+119882(119903119894) (1)

where 1198631199031198941199051198941



119895=1119888119905119894119895119905119894(119895+1)



R1 R2 Rm

Order input

Cost calculated

Task allocation

R


Taskcompletion

Accurate

Centralcontroller

wij cij dij




TT = max119894

ITC (119903119894 119879119894)

TTC =119898

sum

119894=1

ITC (119903119894 119879119894)

BU =min119894ITC (119903

119894 119879119894)

max119894ITC (119903

119894 119879119894)

(2)



















119865 (119909) =MaxTest

TT+TotalTestTTC

+ BU (3)







(9 3 4 8 10 2 1 6 5 7)

(2 5 7 10 9 1 8 4 6 3) (4)


(9 3 4 | 8 10 2 1 | 6 5 7)

(2 5 7 | 10 9 1 8 | 4 6 3) (5)

(9 3 4 8 10 2 1 6 5 7) (2 5 7 10 9 1 8 4 6 3)

(8 10 2 1 5 7 9 4 6 3)(10 9 1 8 3 4 2 6 5 7) Off2Off1






(8 10 2 1 5 7 9 4 6 3)

(10 9 1 8 3 4 2 6 5 7) (7)



(5 3 6 9 2 7 10 1 4 8) (8)


(5 2 6 9 3 7 10 1 4 8) (9)





119905 observe 119903

119905+1 119904119905+1

(10) 119876(119904

119905 119886119905) larr 119876(119904

119905 119886119905) + 120572[119903

119905+1+ 120574max

119886isin119860119876(119904119905+1 119886) minus 119876(119904

119905 119886119905)]

(11) 119904 larr 119904119905+1

(12) if 119904119905+1








119905isin 119878 where 119878 is






119876 (119904119905 119886119905)

larr997888 119876 (119904119905 119886119905)

+ 120572 [119903119905+1+ 120574max119886isin119860

119876 (119904119905+1 119886) minus 119876 (119904

119905 119886119905)]

(10)








120587lowast

(119904) = argmax119886

119876 (119904 119886) (11)



4 Experiments




Picking station Pod







50 100 150 200 250 300 350 4000

1000

2000

3000

4000

5000

6000

7000

8000

9000

Number of tasks

TT




Rand + Alowast

Rand + Q-learning


Number of tasks50 100 150 200 250 300 350 400

010000200003000040000500003000035000400004500050000

TTC









50 100 150 200 250 300 350 4000

102030405060708090

100

BU (

)

Number of tasks










5 Conclusion






Acknowledgment


References






































Volume 2014




Journal of











Journal of


Function Spaces






Algebra























119865 (119909) =MaxTest

TT+TotalTestTTC

+ BU (3)







(9 3 4 8 10 2 1 6 5 7)

(2 5 7 10 9 1 8 4 6 3) (4)


(9 3 4 | 8 10 2 1 | 6 5 7)

(2 5 7 | 10 9 1 8 | 4 6 3) (5)

(9 3 4 8 10 2 1 6 5 7) (2 5 7 10 9 1 8 4 6 3)

(8 10 2 1 5 7 9 4 6 3)(10 9 1 8 3 4 2 6 5 7) Off2Off1






(8 10 2 1 5 7 9 4 6 3)

(10 9 1 8 3 4 2 6 5 7) (7)



(5 3 6 9 2 7 10 1 4 8) (8)


(5 2 6 9 3 7 10 1 4 8) (9)





119905 observe 119903

119905+1 119904119905+1

(10) 119876(119904

119905 119886119905) larr 119876(119904

119905 119886119905) + 120572[119903

119905+1+ 120574max

119886isin119860119876(119904119905+1 119886) minus 119876(119904

119905 119886119905)]

(11) 119904 larr 119904119905+1

(12) if 119904119905+1








119905isin 119878 where 119878 is






119876 (119904119905 119886119905)

larr997888 119876 (119904119905 119886119905)

+ 120572 [119903119905+1+ 120574max119886isin119860

119876 (119904119905+1 119886) minus 119876 (119904

119905 119886119905)]

(10)








120587lowast

(119904) = argmax119886

119876 (119904 119886) (11)



4 Experiments




Picking station Pod







50 100 150 200 250 300 350 4000

1000

2000

3000

4000

5000

6000

7000

8000

9000

Number of tasks

TT




Rand + Alowast

Rand + Q-learning


Number of tasks50 100 150 200 250 300 350 400

010000200003000040000500003000035000400004500050000

TTC









50 100 150 200 250 300 350 4000

102030405060708090

100

BU (

)

Number of tasks










5 Conclusion






Acknowledgment


References






































Volume 2014




Journal of











Journal of


Function Spaces






Algebra















(9 3 4 8 10 2 1 6 5 7)

(2 5 7 10 9 1 8 4 6 3) (4)


(9 3 4 | 8 10 2 1 | 6 5 7)

(2 5 7 | 10 9 1 8 | 4 6 3) (5)

(9 3 4 8 10 2 1 6 5 7) (2 5 7 10 9 1 8 4 6 3)

(8 10 2 1 5 7 9 4 6 3)(10 9 1 8 3 4 2 6 5 7) Off2Off1






(8 10 2 1 5 7 9 4 6 3)

(10 9 1 8 3 4 2 6 5 7) (7)



(5 3 6 9 2 7 10 1 4 8) (8)


(5 2 6 9 3 7 10 1 4 8) (9)





119905 observe 119903

119905+1 119904119905+1

(10) 119876(119904

119905 119886119905) larr 119876(119904

119905 119886119905) + 120572[119903

119905+1+ 120574max

119886isin119860119876(119904119905+1 119886) minus 119876(119904

119905 119886119905)]

(11) 119904 larr 119904119905+1

(12) if 119904119905+1








119905isin 119878 where 119878 is






119876 (119904119905 119886119905)

larr997888 119876 (119904119905 119886119905)

+ 120572 [119903119905+1+ 120574max119886isin119860

119876 (119904119905+1 119886) minus 119876 (119904

119905 119886119905)]

(10)








120587lowast

(119904) = argmax119886

119876 (119904 119886) (11)



4 Experiments




Picking station Pod







50 100 150 200 250 300 350 4000

1000

2000

3000

4000

5000

6000

7000

8000

9000

Number of tasks

TT




Rand + Alowast

Rand + Q-learning


Number of tasks50 100 150 200 250 300 350 400

010000200003000040000500003000035000400004500050000

TTC









50 100 150 200 250 300 350 4000

102030405060708090

100

BU (

)

Number of tasks










5 Conclusion






Acknowledgment


References






































Volume 2014




Journal of











Journal of


Function Spaces






Algebra











119905 observe 119903

119905+1 119904119905+1

(10) 119876(119904

119905 119886119905) larr 119876(119904

119905 119886119905) + 120572[119903

119905+1+ 120574max

119886isin119860119876(119904119905+1 119886) minus 119876(119904

119905 119886119905)]

(11) 119904 larr 119904119905+1

(12) if 119904119905+1








119905isin 119878 where 119878 is






119876 (119904119905 119886119905)

larr997888 119876 (119904119905 119886119905)

+ 120572 [119903119905+1+ 120574max119886isin119860

119876 (119904119905+1 119886) minus 119876 (119904

119905 119886119905)]

(10)








120587lowast

(119904) = argmax119886

119876 (119904 119886) (11)



4 Experiments




Picking station Pod







50 100 150 200 250 300 350 4000

1000

2000

3000

4000

5000

6000

7000

8000

9000

Number of tasks

TT




Rand + Alowast

Rand + Q-learning


Number of tasks50 100 150 200 250 300 350 400

010000200003000040000500003000035000400004500050000

TTC









50 100 150 200 250 300 350 4000

102030405060708090

100

BU (

)

Number of tasks










5 Conclusion






Acknowledgment


References






































Volume 2014




Journal of











Journal of


Function Spaces






Algebra













120587lowast

(119904) = argmax119886

119876 (119904 119886) (11)



4 Experiments




Picking station Pod







50 100 150 200 250 300 350 4000

1000

2000

3000

4000

5000

6000

7000

8000

9000

Number of tasks

TT




Rand + Alowast

Rand + Q-learning


Number of tasks50 100 150 200 250 300 350 400

010000200003000040000500003000035000400004500050000

TTC









50 100 150 200 250 300 350 4000

102030405060708090

100

BU (

)

Number of tasks










5 Conclusion






Acknowledgment


References






































Volume 2014




Journal of











Journal of


Function Spaces






Algebra










50 100 150 200 250 300 350 4000

1000

2000

3000

4000

5000

6000

7000

8000

9000

Number of tasks

TT




Rand + Alowast

Rand + Q-learning


Number of tasks50 100 150 200 250 300 350 400

010000200003000040000500003000035000400004500050000

TTC









50 100 150 200 250 300 350 4000

102030405060708090

100

BU (

)

Number of tasks










5 Conclusion






Acknowledgment


References






































Volume 2014




Journal of











Journal of


Function Spaces






Algebra













Acknowledgment


References






































Volume 2014




Journal of











Journal of


Function Spaces






Algebra




















Volume 2014




Journal of











Journal of


Function Spaces






Algebra
















Volume 2014




Journal of











Journal of


Function Spaces






Algebra









Documents

Research Article Genetic Scheduling and Reinforcement Learning …downloads.hindawi.com/journals/mpe/2015/597956.pdf · 2019-07-31 · Genetic Scheduling and Reinforcement Learning