A Virtual Network Customization Framework for Multicast

IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 38, NO. 6, JUNE 2020 1025

A Virtual Network Customization Framework forMulticast Services in NFV-Enabled Core Networks

Omar Alhussein , Student Member, IEEE, Phu Thinh Do , Member, IEEE, Qiang Ye , Member, IEEE,

Junling Li, Student Member, IEEE, Weisen Shi , Student Member, IEEE, Weihua Zhuang , Fellow, IEEE,

Xuemin Shen , Fellow, IEEE, Xu Li, Member, IEEE, and Jaya Rao, Member, IEEE

Abstract— The paradigm of network function virtualiza-tion (NFV) with the support of software defined network-ing (SDN) emerges as a promising approach for customizingnetwork services in fifth generation (5G) networks. In thispaper, a multicast service orchestration framework is presented,where joint traffic routing and virtual network function (NF)placement are studied for accommodating multicast services overan NFV-enabled physical substrate network. First, we investigatea joint routing and NF placement problem for a single multicastrequest accommodated over a physical substrate network, withboth single-path and multipath traffic routing. The joint problemis formulated as a mixed integer linear programming (MILP)problem to minimize the function and link provisioning costs,under the physical network resource constraints, flow conserva-tion constraints, and NF placement rules; Second, we developan MILP formulation that jointly handles the static embeddingof multiple service requests over the physical substrate network,where we determine the optimal combination of multiple servicesfor embedding and their joint routing and placement config-urations, such that the aggregate throughput of the physicalsubstrate is maximized, while the function and link provisioningcosts are minimized. Since the presented problem formulationsare NP-hard, low complexity heuristic algorithms are proposedto find an efficient solution for both single-path and multipathrouting scenarios. Simulation results are presented to demon-strate the effectiveness and accuracy of the proposed heuristicalgorithms.

Index Terms— 5G networks, SDN, NFV, multicast services,NF chain embedding, MILP, service customization.

Manuscript received January 30, 2019; revised December 3, 2019; acceptedJanuary 28, 2020. Date of publication April 8, 2020; date of current versionMay 21, 2020. This work was supported in part by Research Grants fromHuawei Technologies Canada and in part by the Natural Sciences andEngineering Research Council (NSERC) of Canada. This article was presentedin part at the 2018 IEEE Global Communications Conference (Globecom)2018. (Corresponding author: Omar Alhussein.)

Omar Alhussein, Junling Li, Weisen Shi, Weihua Zhuang, and XueminShen are with the Department of Electrical and Computer Engineering,University of Waterloo, Waterloo, ON N2L 3G1, Canada (e-mail: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]).

Phu Thinh Do is with the Institute of Research and Development, Duy TanUniversity, Da Nang 550000, Vietnam (e-mail: [email protected]).

Qiang Ye is with the Department of Electrical and Computer Engineeringand Technology, Minnesota State University, Mankato, MN 56001 USA(e-mail: [email protected]).

Xu Li and Jaya Rao are with Huawei Technologies Canada Inc., Ottawa,ON K2K 3J1, Canada (e-mail: [email protected]; [email protected]).

Color versions of one or more of the figures in this article are availableonline at http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/JSAC.2020.2986591

I. INTRODUCTION

IN RECENT years, communication networks have experi-enced a radical and fundamental change in the way they are

designed and managed. This shift is mainly due to two impor-tant paradigms, namely software defined networking (SDN)and network function virtualization (NFV). SDN and NFVare considered to be important technical approaches for futurecommunication networks, representing two driving innovationplatforms in the upcoming fifth generation (5G) era. Via NFVand SDN, an approach for establishing service-customizednetworks has a potential to greatly resolve the complexityin conventional networks and to meet new demands and usecases [2]–[7]. A service-customized network provides tradi-tional connectivity between a set of terminals, and allows forthe deployment of abstract network applications in the dataplane. An NFV-enabled network service can be represented bya logical topology, called a network function (NF) chain, whichspecifies a set of virtual NFs that are orchestrated and deployedalong the chosen routes from the source(s) to the destina-tion(s), with some properties and dependencies satisfied. Someexamples of virtual NFs include cache, transcoder, firewall,5G evolved packet core, wireless access network (WAN)optimizer and network address translator (NAT), which canbe hosted and dynamically employed at commodity serversand data center (DC) nodes (called NFV nodes). In thisapproach, the service provider (SP) requests a network serviceto satisfy certain expected demands and requirements as perthe service-level agreement (SLA). In turn, the responsibilityof the service orchestrator or the network operator (NO) isto efficiently design such a virtual network, and place it onthe physical substrate network. The NO aims at reducing theprovisioning cost, and maximizing the utilization of its ownnetworks, while simultaneously meeting the binding SLA.

There is a considerably growing demand for services of mul-ticast nature such as video streaming, multi-player augmentedreality games, and file distribution. For instance, it is estimatedthat 82 percent of the consumers Internet traffic will beattributed to video traffic by 2022 [8]. In addition, the multicastmode of communication can reduce the bandwidth consump-tion in backbone networks by over 50% in contrast to theunicast mode [9]. In an NFV-enabled network, a multicastservice can deploy several NF instances in the data plane,whereas a routing policy forwards the traffic from the source to

0733-8716 © 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: University of Waterloo. Downloaded on July 26,2020 at 19:27:28 UTC from IEEE Xplore. Restrictions apply.

https://orcid.org/0000-0002-1531-5916

https://orcid.org/0000-0002-7623-6242

https://orcid.org/0000-0002-8208-6295

https://orcid.org/0000-0002-0627-7567

https://orcid.org/0000-0003-0488-511X

https://orcid.org/0000-0002-4140-287X

1026 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 38, NO. 6, JUNE 2020

Fig. 1. Illustraion of a multicast NF chain for a basic video streaming service.

each destination while traversing the NFs instances as per theNF chain. Fig. 1 illustrates an example of a multicast NF chainfor a typical video streaming service that distributes content toseveral destinations. If a multicast service requires connectingsome terminal nodes without intermediate NFs, the optimalrouting that reduces the link provisioning cost can be foundby constructing a Steiner tree or one of its variants. A Steinertree is a generalization of the minimum spanning tree (MST)which finds the subset of weighted edges (and nodes) thatconnect all vertices in a graph with the minimum possible linkprovisioning cost. Constructing a Steiner tree is an NP-hardproblem [10], [11]. However, polynomial time approximationalgorithms exist to construct a Steiner tree. With the emergenceof SDN providing a global view of the physical network topol-ogy and network states, Steiner tree-based routing approachesbecome feasible [12]–[14]. However, such approaches do notincorporate NFs in their formulation, and cannot be extendeddirectly to the joint multicast routing and NF placementproblem. Practically, there exist a massive number of NFplacement configurations, each of which requires a multicastrouting topology construction (e.g., one instance of suchconfigurations is shown in Fig. 1). The NF placement andmulticast routing are correlated, which leads to technicalchallenges for orchestrating a single NFV-enabled multicastservice. Selecting just enough NFV nodes for NF placementinevitably increases the link provisioning cost for building anappropriate multicast routing topology; Conversely, instantiat-ing NF instances on more NFV nodes may yield a decreasedlink provisioning cost with traffic load balancing at the expenseof an increased function provisioning cost. Therefore, how tobalance the tradeoff between link and function provisioningcosts is a challenging issue. Our objective is to determine anoptimal NF placement with multicast traffic routing to mini-mize the overall function and link provisioning costs. Anothermajor challenge stems from the fact that multiple network ser-vices share the network substrate. As the network substrate haslimited transmission and processing resources, the efficienciesof embedding multiple service requests interplay. Prioritizinga low-rate network service for embedding can fragment thenetwork resources, thereby hindering other high-rate networkservices from being successfully (or efficiently) embedded.Some recent studies address the orchestration of NFV-enabledmulticast service to minimize the function and link provi-sioning costs [1], [15]–[19]. However, most existing worksassume a design scenario where all NFs are hosted in one NFVnode for each network service and multicast replication pointscan occur only after the deployment of NFs. More realisticand flexible design (e.g., one NFV node is only capable ofhosting specific types of NFs, multipath routing is enabledbetween NFs) can make the existing solutions not feasible or

not optimized. More details of relevant literature are discussedin Section II.

In this paper, we present an optimization framework forthe orchestration of multiple multicast service requests overthe physical substrate. First, we study a joint multicast trafficrouting and NF placement problem for a single service requestto minimize the function and link provisioning costs, underthe physical processing and transmission resource constraints,flow conservation constraints, and NF placement rules. Forpractical applications, our formulated problem focuses on flex-ible multicast routing and NF placement (embedding), wherewe allow one-to-many and many-to-one NF mappings. That is,several NF instances can be hosted at one NFV node if per-missible, and one type of NF can be replicated and deployedon different NFV nodes as NF instances to serve differentsets of destinations, thereby providing considerable flexibilityin the deployment of network services. Furthermore, our for-mulated problem incorporates both single path and multipathtraffic routing between the embedded NF instances. Since theformulated problem is NP-hard, we devise a low-complexityheuristic algorithm to obtain an efficient and flexible solution,based on a key-node preferred minimum spanning tree (MST)approach; Second, we consider a general scenario of placingmultiple service requests over the physical network. Sincemultiple services compete with each other to be hosted ona substrate network with limited resources, we accept theservices which maximize the aggregate throughput with leastprovisioning cost. We formulate an MILP that jointly deter-mines multicast topologies for multiple service requests, wherewe find the combination of network services that maximize theaggregate network throughput while minimizing the overallfunction and link provisioning costs. Moreover, we develop asimple heuristic algorithm that prioritizes the service requests,aiming at maximizing the aggregate throughput with theminimum overall provisioning cost.

The rest of the paper is organized as follows. Section II givesan overview of related work. Section III presents the systemmodel under consideration, which includes the representationof the physical network, NFs, and multicast service requests.Section IV addresses the joint NF placement and routingproblems for multicast services with multipath routing forboth single-service and multi-service cases. Section V presentsMILP formulations for a single-service multipath scenario, anda generalized multi-service multi-path scenario, respectively.Section VI proposes simple modular heuristic algorithms toaddress the complexity of the resultant MILP formulations.Simulation results are presented in Section VII, and conclud-ing remarks are drawn in Section VIII. A list of importantnotations is given in Table I.

II. RELATED WORK

SDN provides a global network view and centralized con-trol over the substrate network topology with the associatednetwork states. Some existing works focus on constructingefficient centralized routing topologies for multicast serviceswithout NFs [12]–[14], [20], which mainly rely on construct-ing a Steiner tree (or one of its variants) to reduce the


ALHUSSEIN et al.: VIRTUAL NETWORK CUSTOMIZATION FRAMEWORK FOR MULTICAST SERVICES 1027

TABLE I

LIST OF IMPORTANT SYMBOLS FOR THE OPTIMAL SINGLE- AND MULTI-SERVICE FORMULATIONS

transmission resource consumption, network state consump-tion, end-to-end (E2E) delay, or to improve the reliability.For instance, Zhao et al. propose an SDN-based video confer-encing solution by constructing a minimum-delay Steiner treewith iterative enhancements [12]. To deal with the increasednetwork state consumption due to multicasting, a Steinertree that jointly minimizes the number of links and branchnodes (with network states) is established, assuming unicas-ting across links with no branches [20]. The state-of-the-artSDN-enabled multicast routing approaches cannot be adoptedor modified directly to incorporate the NF placement.

The virtual NF placement and traffic routing prob-lem has been extensively studied for the unicast servicecase [3], [21]–[28]. For the multicast service case, a simpleapproach is to apply the unicast-based NF placement and rout-ing approaches to each source-destination pair independently,which leads to a waste of network resources with a largeservice provisioning cost. There are relatively few works inthe joint multicast routing and NF placement for multicastservices [1], [15]–[19], where one needs to jointly build amulticast topology and place the NFs. Zhang et al. considerthe joint routing and placement for NFV-enabled multicastrequests, under the assumption that all NFs should be servedby only one NFV node [15], which is extended for a multipleNFV node case under the assumption of an uncapacitatedsubstrate network [16]. It is assumed that multicast replication

points occur only after deployment of NFs in the constructedmulticast topology, which reduces the degree of flexibilityof the orchestration framework. Zeng et al. jointly considerplacement, multicast routing, and spectrum assignment for afibre optical network, where the objective is to minimize thefunction, link, and frequency spectrum provisioning costs [17].The heuristic solution clusters each group of destinations thatshare one specific NF. Then, the solution is divided into twosteps to allocate the NF in an NFV node for each cluster, andto find the MST that spans the source, the allocated NF, and thedestinations. Similarly, it is assumed that, for each multicastservice, the traffic flowing to each destination is processed byone NF.

Xu et al. consider that multiple NFV nodes can host alltypes of NFs [18]. For each source-destination pair, the multi-cast stream needs to pass through only one NFV node whereall NFs are placed before arriving at each destination. Sincethe destinations are distributed in a large area, the algorithmallows activating multiple NFV nodes, where each NFV nodeis responsible for a subset of destinations. However, thereis possibility that some NFV nodes can only host specifictypes of NFs due to hardware-based or subscription-basedrestrictions [17], [22]. A recent work tackles the so-calledservice forest problem for the traffic routing and NF placementof multiple service requests [19]. For a generalized scenariowhere each source-destination pair requires multiple service



requests, the last NF is placed on the physical substrate first,which divides the multicast topology into two parts. The firstpart is from the source to the last NF, while the secondpart from the last NF to all destinations. The first part issolved by the so-called k-stroll algorithm. In the second part,a minimum Steiner tree approximation connects the last NFto all destinations. It is assumed that multicast replicationpoints at the topology occur always after the last NF, andan exhaustive search finds the best place to host the last NFamong all candidate NFV nodes.

To address some research gaps, we aim at developing aflexible multicast routing and NF placement framework forboth single-service and multi-service scenarios. We considerthat multicast replication points can occur before and afterthe deployment of NF instances, thereby providing flexibilityfor topology customization which is particularly crucial forgeographically distributed NF chains (such as in NFV-enabledcore networks). In addition, multipath routing between theembedded NFs is incorporated to improve the utilization ofthe network substrate’s resources.

III. SYSTEM MODEL

A. Physical Substrate Network

Consider a physical substrate network, G = (N ,L), whereN and L are the set of nodes and links. The nodes can beswitches (represented by set F ), commodity servers and datacenter (DC) nodes (namely NFV nodes, represented by setM), i.e., N = F ∪M. Switches are capable of forwardingand replicating traffic, and NFV nodes are capable of hostingand operating NFs. We assume that each NFV node has aforwarding capability, and has available processing rate C(n),n ∈M, in packet per second (packet/s) [29]–[31]. Moreover,an NFV node is capable of provisioning a number of NFssimultaneously as long as the available processing rate satisfiesthe NF processing requirements. Each physical link has alimited transmission rate B(l), l ∈ L, in packet/s.

B. Network Functions

We represent all the NF types by set P , where a specifictype, p ∈ P , resembles some virtual functionality (e.g., IDS,compression, proxy, and LTE packet gateway). We furtherassociate NFV node n (∈ M) with a set of admittable NFtypes using an indicator function kni ∈ {0, 1}, where kni = 1if NFV node n (∈ M) can admit function fi (∈ P).

C. Multicast NF Chains

Let the set of all multicast service requests be denoted byR.A multicast service, r ∈ R, is described by a multicast NFchain, represented by a weighted acyclic directed graph,

Sr = (sr,Dr,Vr, dr), r ∈ R (1)

where sr and Dr represent the source node and the set ofdestinations, Vr = {f r

1 , f r2 , . . . f r

|V|} represents the set offunctions that have to be traversed in an ascending orderfor every source-destination pair, and dr is the data raterequirement in packet/s. Each NF f r

i requires a processing

rate C(f ri ), i ∈ {1, . . . , |Vr|}, in packet/s. An NF instance

belongs to one service request, and it cannot be shared amongmultiple NF chains [15], [16], [19], [21].

IV. PROBLEM DESCRIPTION

We investigate the orchestration of multiple multicast ser-vices over the physical substrate network in two sequentialproblems. In the first problem, joint multipath-enabled multi-cast routing and NF placement are studied for a single service.Given the substrate network and the service description of amulticast NF chain, we intend to design an embedded multi-cast topology for the NF chain. For each source-destinationpair in the embedded topology, all NF types should betraversed in a specified order. Therefore, the first problemconsists of two joint subproblems: (i) NF placement on thephysical substrate, and (ii) multicast traffic routing design fromthe source to the destinations, passing through a sequence ofthe required NFs. Note that multipath routing is enabled forthe paths between embedded NFs to increase the flexibilityof topology customization and improve the physical resourceutilization especially for geographically-dispersed large-scalecore networks. Our objective is to minimize the function andlink provisioning costs in determining an optimal embeddedmulticast topology. However, the minimization of both costsare conflicting. Instantiating a large number of NF instances atmore network locations achieves a balanced traffic load at theexpense of an increased function provisioning cost, whereasfewer NF instantiations reduce the function provisioning costat the expense of less load balancing and inefficient networkresource exploitation. Therefore, it is required to balance thetradeoff to minimize the overall provisioning cost for the NFchain embedding. The first problem is defined as follows:

Problem 1: To determine an optimal multipath-enabledmulticast topology for an NF chain to minimize the functionand link provisioning costs, with all the required NFs traversedin order and with the required processing and transmissionresource constraints satisfied.

In the second problem, we study a joint multicast routingand NF placement problem for a multi-service scenario. NFVallows multiple NF chains to run over a common networksubstrate. However, as the network resources are limited, mul-tiple NF chains may not be accepted on the substrate networksimultaneously. We need to decide which service requestsshould be embedded such that the aggregate throughput ismaximized, while the function and link provisioning costs areminimized. We consider the static NF chain embedding for themulti-service scenario, where all service requests are availablea priori. The online NF placement and routing in which differ-ent types of services arrive and being embedded dynamically issubject to our future research. Therefore, the second problemis defined as follows:

Problem 2: To find an optimal combination of multicast NFchains that maximizes the aggregate throughput of the networksubstrate, while minimizing the respective function and linkprovisioning costs.

We consider that NFs and virtual links have one-to-manyand many-to-one mapping with physical NFV nodes andlinks, respectively, whereby each NF instance can serve a



Fig. 2. Comparison of flexible and non-flexible embedding for a multicastrequest. (a) Topology of NF chain request; (b) Embedding result of NFchain on network substrate with 11 links due to the non-flexible scheme;(c) Modified topology of NF chain request due to the flexible scheme;(d) Embedding result of modified NF chain on network substrate with 10 linksdue to the flexible scheme.

subset of destinations, and the deployment of NF instancescan occur after packet replication in the multicast topology.Such practical consideration achieves higher flexibility andefficiency in the routing and placement processes for largelydistributed networks, since destinations may be geographicallyfar away. Restricting all destinations to share the same setof NF instances can be inefficient for transmission resourcelimited scenarios. When the destinations are geographicallydispersed, an efficient solution is to duplicate and deploythe function instances close to each of the destinations fora reduced link provisioning cost due to flexible routing.We give an illustration in Fig. 2, where the logical topologyof an NF chain request with two NFs and two destinations(i.e., t1, t2 ∈ D) are embedded onto the substrate network.Assume that each physical link can be used only once.With a non-flexible scheme, NFs cannot be replicated andmulticast replication points can exclusively occur after thedeployment of the last NF. Hence, the embedding result of themulticast request (Fig. 2(a)) on the physical substrate requires11 links as shown in Fig. 2(b). With a flexible scheme, the NFchain topology is modified, as shown in Fig. 2(c), where therespective embedding result on the physical substrate requires10 links as shown in Fig. 2(d).

V. PROBLEM FORMULATION

We first present the problem formulation for a single-servicemultipath scenario, followed by a generalized multi-servicemultipath scenario.

A. Single-Service Multipath Scenario

To establish a joint multipath-enabled multicast routing andNF placement framework for a single-service, let Jr denotethe maximum number of multicast trees to deliver multicast

Fig. 3. Two Steiner trees that share the same source, traversed functions,and destinations.

service r (∈ R) from the source to the destinations.1 Eachtree emanates from the source and passes by the same set oftraversed NFV nodes and destinations. Fig. 3 illustrates onemulticast request with three destinations and three functions.We use two trees (i.e., Jr = 2) to support up to two multipathrouting paths. If f1 is embedded on node 2 for both trees,we have multipath routes from source node s to node 2, andthe two trees converge afterwards. When Jr = 1, the problemreduces to the single-path routing case. The maximum numberof multicast trees (resembling the maximum possible numberof multipath routes) Jr is an input to the problem formulation.The formulation allows the multicast trees to overlap witheach other, and can be deactivated if needed. In what follows,we describe the details of the MILP formulation for thesingle-service scenario.

Let Snm denote the set of integers from m to n (> m),

i.e., Snm � {m, m + 1, . . . , n} with m, n ∈ Z+. Define binary

variable xjrli ∈ {0, 1}, where xjr

li = 1 indicates that link l(∈ L) is used for forwarding traffic for service r in multicasttree j from f r

i to f ri+1 where i ∈ S|V

r|−11 ; Binary variable

xjrl0 = 1 indicates that link l carries traffic from sr to f r

1 ,and xjr

l|V| = 1 indicates that link l carries traffic from f r|V|

to any destination node t (∈ Dr); Define yjrlit ∈ {0, 1}, where

yjrlit = 1 indicates that link l is used to direct traffic for service

r in multicast tree j from f ri to f r

i+1 for destination t (∈ Dr);Binary variable yjr

l0t = 1 indicates that link l is used to directtraffic for service r in tree j from sr to f r

1 for destination t,and yjr

l|V|t = 1 indicates that link l directs traffic for service rin tree j from f r

|V| to destination t.

With the definitions of x = {xjrli } and y = {yjr

lit}, we have

yjrlit≤xjr

li , l∈L, i∈S|Vr|

0 , j∈SJr

1 , t∈Dr , r∈R. (2)

Furthermore, define binary variables zrni ∈ {0, 1}, where

zrni = 1 indicates that an instance of f r

i is deployed on NFVnode n for service r where i ∈ S|V|

1 , and urnit ∈ {0, 1}, where

1Note that the superscript r which is used in the single-service formulation(subsection V-A) can be dropped since |R| = 1. However, it is necessaryas we develop the MILP formulation for the multi-service scenario insubsection V-B.



urnit = 1 indicates that an instance of f r

i is deployed on n forservice r for destination t. Similarly, we have a relationshipconstraint between z = {zr

ni} and u = {urnit} as

urnit ≤ zr

ni, n ∈ N , i ∈ S|Vr |

1 , t ∈ Dr, r ∈ R. (3)

For each service r (∈ R), we build Jr multicast trees toexploit the multipath property, where each tree can providea fractional transmission rate djr of the overall requiredtransmission rate dr. Therefore, to meet the total requiredtransmission rate dr, we impose the constraint

Jr∑j=1

djr ≥ dr, r ∈ R. (4)

Next, we incorporate the routing and placement constraintin (5) to ensure that traffic flows pass from the source tomultiple destinations through the NF chain. Let f r

0 and f r|V|+1

denote dummy functions (without processing requirements)that are placed on the source node sr and each destination nodet (∈ Dr), respectively. In our model, some of the multicasttrees can be deactivated if needed. Consequently, we have∑

(n,m)∈L yjr(n,m)it −

∑(m,n)∈L yjr

(m,n)it

=

{ur

nit − urn(i+1)t, tree j is activated

0, otherwise(5)

for n ∈ N , i ∈ S|Vr|

0 , t ∈ Dr, r ∈ R, where

urs0t = 1, ur

n0t = 0, ∀t ∈ Dr, n ∈ N\{sr}ut(|Vr|+1)t = 1, un(|Vr|+1)t = 0, ∀t ∈ Dr, n ∈ N\Dr

ursit = 0, ur

tit = 0, ∀i ∈ S|Vr |

1 , t ∈ Dr.

Define binary variable πjr ∈ {0, 1}, where πjr = 1 indicatesthat tree j of service r is activated. Consequently, all variablesrelated to deactivated trees (i.e., xjr

li , yjrlit, and djr) should be

forced to zero. Therefore, we impose the constraint

xjrli ≤ πjr , yjr

lit ≤ πjr , djr ≤ πjrdr,

l ∈ L, i ∈ S|Vr|

0 , j ∈ SJr

1 , t ∈ Dr, r ∈ R. (6)

To guarantee that the routing and placement constraint isconsidered only when tree j of service r (∈ R) is activated,we modify (5) to∑(n,m)∈L

yjr(n,m)it −

∑(m,n)∈L

yjr(m,n)it = πjr

(ur

n(i+1)t − urnit

),

n ∈ N , i ∈ S|Vr |

0 , t ∈ Dr, r ∈ R. (7)

Since we have yjrlit ≤ xjr

li in (2), the constraint yjrlit ≤ πjr

in (6) can be removed. Thus, we re-write (6) as

xjrli ≤ πjr , djr ≤ πjrdr, l ∈ L, i ∈ S|V

r |0 , j ∈ SJr

1 . (8)

Constraint (8) means that we consider xjrli and djr when tree

j is activated; otherwise, we simply set these variables to zero.We require that exactly one instance of function f r

i is traversedfor every source-destination pair, which can be expressed as∑

n∈M urnit = 1, i ∈ S|V

r |1 , t ∈ Dr, r ∈ R. (9)

Moreover, function f ri is hosted at node n only when

admittable and when the resources at node n are sufficient.We have

∑r∈R

|Vr|∑i=1

zrniC(f r

i ) ≤ C(n), n ∈M, i ∈ S|Vr|

1 (10a)

zrnikni = 1, n ∈M, i ∈ S|V

r |1 , r ∈ R (10b)

where kni = 1 indicates that node n can admit function fi;otherwise, kni = 0.

Objectives — Following the relevant research onNFV-enabled service provisioning [17]–[19], [21]–[23],we aim to minimize the function (processing) andlink (transmission) provisioning costs over all Jr multicasttrees for service r, in addition to balancing the substratenetwork resources in the long run as

min α∑l∈L

Jr∑j=1

|Vr|∑i=0

(djr

B(l)+1

)xjr

li + β

|Vr|∑i=1

∑n∈M

C(f ri )

C(n)zr

ni.

(11)

In (11), we minimize the weighted sum of the cost of forward-ing the traffic over the utilized physical links of the substratenetwork for all activated trees, and the cost of provisioningthe NF instances in the NFV nodes. Parameters α and β arethe weighting coefficients to reflect the importance level ofminimizing the cost of traffic forwarding and minimizing thecost of NF provisioning respectively, where α + β = 1 andα, β > 0. The terms djrxjr

li /B(l) and C(f ri )zr

ni/C(n) ensureload balancing over the physical links and NFV nodes [32].Moreover, the term ‘+1’ in (11) minimizes the number oflinks in building the multicast topology. Denote the productterm djrxjr

li in the objective function (11) by γjrli as

γjrli = xjr

li djr , l ∈ L, i ∈ S|Vr |

0 , j ∈ SJr

1 , r ∈ R. (12)

The term, γjrli , can be interpreted as the transmission rate over

link l to deliver traffic from f ri to f r

i+1 in tree j. The aggregaterate from all services over link l is upper bounded by theavailable link transmission rate B(l), i.e.,

∑r∈R

Jr∑j=1

|Vr|∑i=0

γjrli ≤ B(l), l ∈ L. (13)

In summary, the optimization problem for the single servicescenario is formulated as

(P1) : min α∑l∈L

Jr∑j=1

|Vr|∑i=0

( γjrli

B(l)+ xjr

li

)

+ β

|Vr|∑i=1

∑n∈M

C(f ri )

C(n)zr

ni (14a)

subject to (2)− (4), (7)− (10), (12), (13) (14b)

x, y, z, u, π ∈ {0, 1}, d � 0, γ � 0. (14c)

In (14), the objective function and all constraints are lin-ear except constraints (7), (12), (10b). In the next step,we transform these non-linear constraints to equivalent linearconstraints such that a standard MILP solver can handle them.



To do so, for nonlinear constraint (7), the bilinear term πjrurnit

can be handled by introducing a new variable wjrnit = πjrur

nit.Constraint (7) is changed to∑

(m,n)∈Lyjr(m,n)it −

∑(n,m)∈L

yjr(n,m)it = wjr

nit−wjrn(i+1)t,

n∈N , i∈S|Vr |

0 , t∈Dr , j∈SJr

1 , r∈R. (15)

The corresponding relations among wjrnit, πjr , and ur

nit aregiven by

wjrnit ≤ πj , wjr

nit ≤ urnit, wjr

nit ≥ πjr + urnit − 1,

n ∈ N , i ∈ S|Vr|

0 , t ∈ Dr, j ∈ SJr

1 , r ∈ R. (16)

For nonlinear constraint (10b), denote the product term zrnikni

by grni. Consequently, (10b) can be expressed by

grni ≤ zr

ni, n ∈M, i ∈ S|Vr|

1 (17a)

grni ≤ kni, n ∈ M, i ∈ S|V

r|1 (17b)

grni ≥ zr

ni + kni − 1, n ∈ M, i ∈ S|Vr |

1 . (17c)

For nonlinear constraint (12), we utilize the big-M method,and express it equivalently as

djr−M(1−xjrli ) ≤ γjr

li ≤ djr,

l ∈ L, i ∈ S|Vr|

0 , j ∈ SJr

1 , r ∈ R (18a)

0 ≤ γjrli ≤Mxjr

li ,

l ∈ L, i ∈ S|Vr|

0 , j ∈ SJr

1 , r ∈ R (18b)

where M is a large positive number. Since djr is upperbounded by dr, γjr

li given by (12) is bounded above by dr.Thus, it suffices to set M = dr.

As a result, the non-linear optimization problem fora single-service in (14) can be re-written in an MILPform as

(P1�) : minX

∑l∈L

Jr∑j=1

|Vr|∑i=0

α

(γjr

li

B(l)+ xjr

li

)

+ β

|Vr|∑i=1

∑n∈M

C(f ri )

C(n)zr

ni (19a)

subject to (2)−(4), (8)−(10a), (13), (14c)− (18)(19b)

where X = {x, y, z, u, w, π, d, γ}, and (19) can be solvedby an MILP solver.

B. Multi-Service Multipath Scenario

In this subsection, we consider the scenario of jointlyhandling multiple multicast service requests. We formulatean MILP that jointly constructs the multicast topology formultiple service requests, where the goal is to find the combi-nation of service requests such that the aggregate throughputis maximized while the overall function and link provisioningcosts are minimized. First, we need to maximize the achievedaggregate throughput obtained by hosting network services on

the substrate network. The achieved aggregate throughput isgiven by

R =∑r∈R

Rrρr, (20)

where ρr ∈ {0, 1} is a binary decision variable withρr = 1 when service r is accepted, and Rr is the correspond-ing throughput, defined as

Rr = a1

|Vr|∑i=1

C(f ri ) + a2

(|Vr|+ |Dr|

)dr, r ∈ R (21)

where the first and second terms denote the requiredamount of processing and transmission rates, respectively,in packet/s [29]–[31]. The two parameters, a1 and a2, areused to tune the priority of processing and transmission ratesrespectively, where a1 + a2 = 1 and a1, a2 > 0.

Second, in addition to maximizing the achieved aggregatethroughput, an efficient configuration to utilize the resourcesefficiently is needed. Specifically, there can be multiple solu-tions to maximize the aggregate throughput defined above.Among such solutions, we find the configuration with the leastfunction and link provisioning cost to save resources for futureservices. The multi-service scenario is cast as a two-step MILP,the first step aims to find the maximum achievable aggregatethroughput, followed by a formulation which finds the routingand NF placement for each admitted NF chain subject to themaximum achievable aggregate throughput.

In the multi-service scenario, some of the service requestscan be rejected due to limited resources. Therefore, we firstgeneralize some of the previous constraints as follows. Con-straint (4) is generalized to

Jr∑j=1

djr ≥ ρrdr, r ∈ R (22)

where the summation of the fractional transmission rate fromall trees for service r (∈ R) is forced to zero when the serviceis rejected (i.e. when ρr = 0). Moreover, an instance of f r

i

is deployed at only one NFV node if service r is accepted,i.e., (9) becomes

∑n∈M

urnit = ρr, i ∈ S|V

r |1 , t ∈ Dr, r ∈ R. (23)

Similarly, when service r is rejected, all variables related tothe service should be zero, i.e.,

πjr≤ρr, zrni≤ρr, n∈N , i∈S|V

r|1 , j∈SJr

1 , r∈R. (24)

Objectives — The objective of the first step is to findthe maximum aggregate throughput R∗. Then, we aim tominimize the function and link provisioning costs for alladmitted services, subject to the maximum achievable aggre-gate throughput R∗, in the second step.



Now we present the first step of maximizing the aggregatethroughput as

(P2) : maxx,y,z,u,ρ,π,w,d,r

∑r∈R

Rrρr (25a)

subject to (2), (3), (8), (10a), (13), (15) (25b)

(16)− (18), (22)− (24) (25c)

x, y, z, u, ρ, π, w∈{0, 1}, d, r≥0. (25d)

After solving (25), we obtain a configuration that provide amaximal aggregate throughput, R∗, for the given |R| servicesand substrate network. However, such configuration can beone among many that can yield R∗. Among all possibleconfigurations, we choose one such that the function and linkprovisioning costs are minimized.

Therefore, in the second step, we find the combinationof admitted services and their multicast topologies such thatthe function and link provisioning costs are minimized, sub-ject to the maximum achievable aggregate throughput R∗,as follows,

(P3) : minx,y,z,u,ρ,π,w,d,r

∑r∈R

∑l∈L

∑j∈SJr

1

|Vr|∑i=0

α

(γjr

li

B(l)+xjr

li

)

+∑r∈R

|Vr|∑i=1

∑n∈N

βC(f r

i )C(n)

zrni

(26a)

subject to (2), (3), (8), (10a), (13), (15) (26b)

(16)− (18), (22)− (24) (26c)

x, y, z, u, ρ, π, w∈{0, 1}, d, r≥0 (26d)∑r∈R

Rrρr ≥ R∗. (26e)

The problem in (26) is an MILP, and can be solved optimallyby a standard MILP solver. After solving (26), we obtainoptimal solutions such that the maximal aggregate throughputR∗ is achieved with minimal function and link provisioningcosts.

Remark: The joint multicast routing and NF placementproblems for both single-service and multi-service cases areNP-hard.

Proof: We first show that our single-service problem (P1)can be reduced from the Steiner tree problem in polynomialtime. Assume a service request with only one function (f ) andmultiple destinations (D). We have a physical substrate (G)such that the source (s) is a leaf vertex (in G) connected tothe only feasible NFV node for f . The optimal solution canbe obtained in two steps. First, we build a Steiner tree fromthe NFV node to the destinations. Second, we place f on theNFV node, and connect the NFV node to the source (as thisis the only feasible option). The first step is NP-hard, whilethe second step is performed in polynomial time. Thus, (P1) isNP-hard. It is then proved that the multi-service problem (P3)is NP-hard as it includes (P1) as a special case [33], [34].

VI. HEURISTIC ALGORITHMS

Even though (P1�), (P2), and (P3) in Section V canbe solved optimally by an MILP solver, the computationaltime is high. A low-complexity heuristic algorithm is neededto efficiently find a solution. The proposed framework ismodular in its design, and can be divided into two mainsteps. First, a mechanism is employed to prioritize servicerequests based on some heuristics that aim to maximize theaggregate throughput. Second, the prioritized service requestsare embedded sequentially using the joint placement androuting (JPR) algorithm for a single-service.

A. Joint Placement and Routing for Single-Service Scenario

We design a single-service heuristic algorithm based onthe following considerations: (i) Different types of NFs canrun simultaneously on an NFV node; (ii) The traversedNF types and their order should be considered for eachsource-destination (S-D) pair; (iii) The objective is to minimizethe provisioning cost of the multicast topology based on (11).According to the aforementioned principles, a two-step heuris-tic algorithm is devised as follows: We first minimize thelink provisioning cost by constructing a key-node preferredMST (KPMST)-based multicast topology that connects thesource with the destinations; Then, we greedily perform NFplacement such that the number of NF instances is minimized.The pseudo-code of the JPR heuristic is shown in Algorithm 1,which is explained in more detail as follows.

First, we copy the substrate network G into G�. Second,to prioritize the NFV node selection in building the KPMST,we calculate new link weights for G� as

ωl′ = α

(dr

B(l�)+ 1

)+ β

dr

C(m),

l� = (n, m) ∈ L�, L� ⊆ G�, r ∈ R (27)

where C(m) is set to a small value when m is a switch; oth-erwise, it represents the processing resource of NFV node m.Then, a key-NFV node is selected iteratively. We constructthe metric closure of G� as G��, where the metric closureis a complete weighted graph with the same node set Nand with the new link weight over any pair of nodes givenby the respective shortest path distance. From the metricclosure, we find the MST which connects the source node,destination nodes, and the key-NFV node. An initial multicastrouting topology (Gv) can be constructed by replacing theedges in the MST with corresponding paths in G� whereverneeded. We then greedily place the NFs from the source ofthe multicast topology to its destinations with the objective ofminimizing the number of NF instances. The cost C(Gv) ofthe new multicast topology (as in (11)) and the number A(Gv)of successfully embedded NF instances are computed. Theobjective is to jointly maximize the number of successfullyplaced NFs and minimize the provisioning cost by iteratingover all candidate key-NFV nodes. In every iteration, a newkey-NFV node is selected. If A(Gv) is increased, we updatethe selected multicast topology with the new key-NFV node;If A(Gv) is unchanged and C(Gv) is reduced, we also update



the selected multicast topology. If a path cannot accommo-date all required NFs (i.e., f1, f2, . . . , f|V|) after selecting akey-NFV node, we devise a corrective subroutine that placesthe missing NF instances on the closest NFV node from themulticast topology, and the corresponding physical links arererouted as follows. Let Pt be the path from s to t in Gv,Pt,f be the union of the paths such that a missing function(f ) is not hosted

(i.e.,{Pt,f = ∪t∈DPt| f is not hosted}

),

and Pc,t,f be longest common path before first branch in P .Correspondingly, for each missing NF, we link the nearestapplicable NFV node to Pc,t,f , place the missing NF instance,and remove all unnecessary edges.

So far, a flexible multicast topology that connects thesource with the destinations, with all NF instances traversedin order, is constructed. We first resume from the single-pathscenario (J = 1) to check whether each path in Gv satisfiesthe link transmission rate requirement as per (13), and findan alternative path for each infeasible path. If a single-pathsolution is infeasible due to (13), the heuristic algorithm forthe single-path case is extended to the multipath-enabled NFchain embedding problem (with J > 1). Enabling multipathrouting provides several advantages. It is activated whenthe transmission rate requirement between two consecutivelyembedded NF instances cannot be satisfied (i.e., when B(l) <dr, l ∈ L); and multipath routing is to reduce the overall linkprovisioning cost compared with the single-path case. For eachpath between two embedded NF instances that does not satisfythe transmission rate requirement dr, we increment the numberof multipath routes gradually, and look for a feasible solutionup to the predefined J . The algorithm declares that the probleminstance is infeasible when the number of multipath routesexceeds J . For J > 1, we trigger a path splitting mechanismfor each path between two embedded NF instances for eachS-D pair as follows.

Let W t,ki,i+1 be the kth path between two embedded NFs

(fi,fi+1) along the network substrate for destination t (∈ D),where the cardinality of all such possible paths is Kt

i,i+1.We first rank all candidate paths in a descending orderbased on the amount of residual transmission resource. Then,we sequentially choose the paths from the candidate paths,such that the summation of all chosen paths’ residual transmis-sion rate meets the required transmission rate dr. Assuming thecurrent number of trees is j, the transmission rates allocatedon the kth path (W t,k

i,i+1) is then calculated as

R(W t,ki,i+1) =

Bkmind

r∑min{j,Kti,i+1}

k=1 Bkmin

,

t ∈ D, i ∈ S|V|0 , k∈Smin{j,Kt

i,i+1}1 , r∈R (28)

where Bkmin is the amount of minimum residual transmission

resources for path W t,ki,i+1, i.e., Bk

min = minl∈W t,ki,i+1

B(l).Here, the multipath extension method essentially computes alink-disjoint multipath configuration from a single-path route.Therefore, the proposed multipath extension is necessarilyprone to the so-called path diminuition problem, in which notall end-to-end multipath-enabled configurations can be devisedfrom a single-path discovery [35], [36].

Future research is needed to incorporate the end-to-end(E2E) delay requirement to achieve quality of service satis-faction. This is a challenging issue since directly expressingthe E2E delay as a function of the decision variables of an opti-mization problem in a closed form is cumbersome. However,given an embedded NF chain, one can model (or measure)the E2E delay, upon which an (iterative) algorithm can bedeveloped to re-adjust the embedding solution of violated NFchains. For instance, in our previous work [37], we proposean analytical E2E packet delay modeling framework, based onqueueing network modeling, for multiple embedded NF chainswhile taking into account the computing and transmissionresource sharing. By incorporating the proposed E2E delaymodel, we will investigate how to develop a delay-aware NFchain embedding algorithm in our future work.

Algorithm 1 Heuristic Algorithm for the Joint NF Place-ment and Routing

1 Procedure JPR (G, Sr);Input : G�(N �,L�), Sr = (sr,Dr, f r

1 , f r2 , . . . , fr

|V|, dr)

Output: Gv

2 Cr ←∞; Ar ← 0;3 for n ∈ M do4 G�� ← MetricClosure(G�, {n, s,D});Gtemp

v (Nv,Lv)← KruskalMST(G��);5 for path from s to each t ∈ D do place NFs from V

on available NFV nodes sequentially subject to (10);6 if A(Gt

v) = Ar & C(Gtv) < Cr then Gv ← Gt

v;Cr ← C(Gt

v);7 else if A(Gt

v) > Aref then Gv ← Gtv;

Aref ← A(Gtv); Cref ← C(Gt

v);8 end9 for each missing NF (f ) from Gv do link nearest NFV

node that can host f to Pc,t,f , and remove unnecessaryedges;

10 for path from fi to fi+1 for each t in Gv do11 success ← false;12 for j = 1 : J do13 Find temp = min{Kt

i,i+1, j} paths from G;14 if

∑tempk=1 minl∈W t,k

i,i+1B(l) ≥ dr then allocate

transmission resource for each kth path (W t,ki,i+1)

using (28); success ← true; break;15 end16 if success = false then break;17 end18 if success = false then Gv ← none;19 else return Gv;

B. Multi-Service Scenario

To achieve high throughput and to efficiently utilize thenetwork resources, our key strategy is to selectively prioritizethe network services contributing significantly to the aggregatethroughput with least provisioning cost. Here, we consider astatic algorithm, i.e., service requests are available a priori.Three principles serving as criteria to prioritize each service



for embedding are identified. The first principle is to rank thegiven network services based on the aggregate throughput,which is defined in (21). A network service with higherachieved throughput has higher priority to be embedded in thesubstrate network, since such service contributes more to theachieved aggregate throughput than a lower ranked service.However, ranking a service based on the throughput alonedoes not take into account the impact of the provisioningcost. It is impossible to obtain the exact provisioning cost ofembedding a service request prior to the routing and placementprocess, as the problem itself is NP-hard. However, the relativepositions of source and destinations can provide hints onthe cost necessitated to host such service. Given a networkservice with the destinations far from each other (i.e., highlydistributive), the provisioning cost is large since more physicallinks and multicast replication points are expected to connectthe destinations. Moreover, the distance from the source todestinations is proportional to network service’s provisioningcost as a long routing path with a relatively large number ofNF instances is needed to establish the multicast topology.

Combining both effects of the distances between destina-tions and the distance from source to destinations, we definea distribution level, denoted by gr, as the product of twocomponents. The first component is Ar/A, where A is thearea of the smallest convex polygon that spans all nodes in thenetwork, and Ar is the area of the smallest convex polygon thatspans all destinations of service r. The ratio Ar/A providesan estimate of how dense a set of destinations of one serviceis in a given area of the network. Note that existing algorithmsto determine the convex hull of a set of points and to calculatethe area of an arbitrary shape are available [38]. The secondcomponent is qr/q, where qr is the distance from source tothe center point of the set of destinations in service r, andq is the largest distance between two arbitrary nodes in thesubstrate network. The center point of the set of destinationsin one service plays a role as a representer for all destinationsin that service. The ratio qr/q can measure how far is thesource from the destinations. The distribution level metric gr

is thus expressed as

gr =Arqr

Aq, r ∈ R. (29)

A larger value of gr implies a higher distribution level, wherethe source is positioned farther away from its destinations andthe destinations are more distribution in the whole networkcoverage area. A largely distributive network service consumesmore network resources, resulting in a high provisioning cost.Therefore, a service with a lower value of gr has a higherpriority to be embedded in order to preserve the substratenetwork resources. Although the parameters in (29) can becalculated with regard to the hop-count (e.g., via computingthe shortest path) which is a more representative metric thanthe Euclidean distance, it is exhaustive to do so.

Next, we introduce a third metric named size to incorporateboth (21) and (29) as follows,

U r = Rr(1− gr), r ∈ R (30)

where the goal is to prioritize a service with higher throughputsubject to a correction factor of 1 − gr for how distribu-tive (costly) it is.

To summarize, we calculate the throughput, the distributionlevel, and the size for each service request using (21), (29),and (30). Then, service requests are sorted according to theirsizes in a descending order, and embedded using Algorithm 1.

C. Complexity Analysis

First, the heuristic algorithm for the single-service scenario(Algorithm 1) iterates over |M| NFV nodes to find a key-NFVnode. For each potential key-NFV node, a multicast topologyis constructed and compared with the previous iteration (inLines 3-8). Denote the set of destinations, source node, and thekey NFV node by T (i.e., T = {n, s,D}). For each potentialkey-NFV node, we construct the metric closure on T , which iscomputed by considering the all-pairs shortest-path algorithmon T . Thus, the worst-case running time is O(|N ||T |2). Then,we find an MST on the constructed metric closure. The MSTis transformed to the Steiner tree by replacing each edge withthe shortest path, and removing unnecessary edges (Line 4).The worst-case time complexity of the MST-based Steinertree is dominated by the metric closure. The constructionof the Steiner tree is followed by an NF placement processthat requires O{|D||M|} time in the worst-case since a pathfrom the source to each destination passes by at most |M|NFV nodes (Line 5). To extend to the multipath scenario,in Lines 10-17, for each path between two embedded NFinstances, we find up to J paths and split the traffic accord-ing to (28), which requires O(J |D| |V| |N | log |N |) time inthe worst-case. For the multi-service scenario, we measurethe size of each request, followed by a sorting algorithmbased on the size in (30), which requires O(|R| log |R|)time. Consequently, each service request is embedded sequen-tially. Hence, the overall worst-case time complexity ofthe multi-service multi-path scenario is O

(|R| log |R| +

|N ||D|(|D| + J |V| log |N |)).

VII. SIMULATION RESULTS

In this section, simulation results are presented to evaluatethe optimal and heuristic solutions to the joint multicastrouting and NF placement problems for the single- andmulti-service scenarios, considering both single-path and mul-tipath routing cases. The simulated substrate network is amesh-topology based network [39], with |N | = 100 and|L| = 684, as shown in Fig. 4. We randomly choose 25 verticesas NFV nodes in the mesh network, and the transmission rateof physical link l and the processing rate of NFV node nare uniformly distributed between 0.5 and 2 Million packet/s(Mpacket/s), i.e., B(l), C(n) ∼ U(0.5, 2) Mpacket/s. To solvethe formulated MILP problems, we use the Gurobi solverwith the branch and bound mechanism, where the weightingcoefficients are set as α = 0.6, β = 0.4. The processing raterequirement of the NFs are set to be linearly proportional tothe incoming data rate, i.e., C(f r) = dr [40].

First, we conduct a comparison between the optimal solutionof the single-service single-path MILP formulation and the



Fig. 4. Mesh topology with |N | = 100 and |L| = 684.

Fig. 5. Embedding cost with respect to (a) the number of destinations and(b) the number of required NFs.

solution of the heuristic. We generate random service requestswhere the numbers of NFs and destinations are varied from3 to 14 and 2 to 11, respectively. The data rate of the generatedservice requests are set to dr = 0.2 Mpacket/s. The totalprovisioning cost obtained from both optimal and heuristicsolutions are shown in Fig. 5, as the number of destinations |D|or NFs |V| increases. It can be seen that the total provisioningcost increases with |D| or |V|. As |D| increases, the costsobtained from both optimal and heuristic solutions grows withnearly a constant gap. Adding destinations incurs a higher costthan adding NF instances, since additional physical links andNF instances are required for each destination.

Fig. 6. Comparison of embedding cost between the proposed JPR heuristicalgorithm and HA-TAA heuristic algorithm in [16].

Next, we compare the proposed heuristic algorithm (JPR)with the heuristic algorithm named HA-TAA2 in [16]. Fora fair comparison, we consider only the single-path scenario.To add heterogeneity to NFV nodes, among the 25 NFV nodesand up to 6 types of NFs, each NF type can be hosted in anNFV node with the chance of 80%. We randomly generate10 multicast requests that have equal number of NFs anddestinations (i.e., |V| = |D|), and each multicast requestis embedded on 30 network substrate instances to reducethe effect of randomness. As shown in Fig. 6, the proposedalgorithm consistently outperforms HA-TAA when |V| =|D| > 2. However, the average running time for the modifiedHA-TAA algorithm was around 0.115 seconds, whereas ourJPR heuristic algorithm took about 2.110 seconds due to thehigher time complexity of the involved algorithm. In [16],the HA-TAA algorithm first finds the shortest path from thesource to the NFV nodes that can host the required NFssequentially, and the shortest path from the last NFV nodeto the respective closest destination. Second, it connects thedestinations through an MST. Since the placement of each NFis only dependent on the previous NF, the HA-TAA algorithmmay take a long path to place NFs before connecting thelast NFV node to the closest destination. In our algorithm,we first focus on finding a Steiner tree from the source tothe destinations through a key-NFV node, thereby optimizingthe link provisioning cost. Then, we modify some of the linksto host all required NF instances (if needed). Here, NFs canbe duplicated to serve different sets of destinations, therebyproviding more flexibility and reduced overall provisioningcost. When |V| ≤ 2, the HA-TAA is specifically more efficientas the NF placement is related to the locations of both thesource and the closest destination.

Fig. 7 shows the advantage of multipath routing oversingle-path routing using the proposed optimal formulation.We depict the maximum supported data rate dr, with whichthe NF embedding is feasible. Compared to the single-path

2The heuristic algorithm (HA-TAA) in [16] assumes uncapacitated networksubstrate. We modify the algorithm to the capacitated scenario, change theweights to match our objective function, and assume that an NFV node canhost multiple NF instances subject to processing resources. The objectivefunction excludes the minimization of number of hops for fairness.



Fig. 7. Maximum supported data rate (d r) for both single-path and multipathrouting scenarios using the proposed optimal formulation.

Fig. 8. Mesh topology with |N | = 100, |L| = 684, and 4 access regionsand 1 core network region.

routing case, multipath routing (J = 2) always supportshigher or equal data rates. With an increase of the numberof required NFs, the maximum supported data rate decreases,and converges to the single-path scenario; the processing costbecomes more significant and the number of candidate NFVnodes and paths decrease.

Next, we demonstrate the effectiveness of the proposedheuristic admission mechanism for the multi-service scenariodiscussed in Subsection VI-B. First, we divide the mesh topol-ogy into four access network regions and one core networkregion as indicated in Fig. (8). Three scenarios with differentavailable processing and transmission rates on the NFV nodesand physical links are considered in the scale of Mpacket/s,as listed in Table II. Each service randomly originates from oneaccess network region, traverses the core network region, andterminates in one of the other three access network regions.For each network scenario, to simulate network congestion,35 multicast service requests are randomly generated andsubmitted for embedding, where the data rate dr of servicer is randomly distributed between [1.5, 3.5] Mpacket/s, andthe number of NFs and destinations are randomly generated

TABLE II

PROCESSING AND TRANSMISSION RATES

Fig. 9. Aggregate throughput comparison of the random admission and theproposed heuristic admission.

Fig. 10. Acceptance ratio comparison of the random admission and theproposed heuristic admission.

as |Vr| = {3, 4} and |Dr| = {3, 4, 5}. For each scenario,the service generation and embedding are repeated 5 timesto reduce the effect of randomness. As a benchmark, we usea second strategy that randomly selects the service requestsfor embedding.

Fig. 9 shows the aggregate throughput R as in (20) achievedby the random and heuristic admission solutions under threescenarios specified in Table II, which increases as the avail-able processing and transmission rates increase. As shown,the aggregate throughput of the proposed heuristic solutionexceeds the aggregate throughput of the random admission by15.63% on average over all the scenarios. This is becausethe size metric used in the heuristic solution ensures thatthe services with a larger throughput are embedded with ahigher priority. Fig. 10 shows the acceptance ratio of thetotal 35 service requests under different average data rates.



Fig. 11. Resource utilization ratio comparison of the random admission andthe proposed heuristic admission.

The acceptance ratio of the heuristic solution exceeds therandom solution over all source data rate levels by 4% onaverage.

Fig. 11 compares the normalized resource utilizationbetween the random and heuristic admission solutions. Theutilization ratio is calculated as the amount of resourcesconsumed by all embedded services over the total availableresources of the substrate network. We normalize the mea-sured utilization ratio by the utilization ratio of the proposedheuristic admission to highlight the enhancement broughtby the heuristic solution. In scenarios 1 and 2, the heuris-tic solution increases the utilization ratio by approximately12.62% and 7.97% over that of the random admission solution,respectively. In scenario 3, the difference in the utilizationratio between the two solutions decreases, especially the linktransmission usage. Recall that the heuristic scheme aimsto maximize the aggregate throughput as defined in (20).Therefore, although in scenario 3 the utilization ratio achievedby the heuristic scheme is close to that of the random scheme,the aggregate throughput achieved by the former is larger asshown in Fig. 9.

VIII. CONCLUSION

In this paper, we study a joint traffic routing and NFplacement framework for multicast services over a substratenetwork under an SDN-enabled NFV architecture. Within theframework, joint multipath-enabled multicast routing and NFplacement is investigated first for a single-service scenarioand then for a multi-service scenario. For the single-servicescenario, an optimization problem is formulated to mini-mize function and link provisioning cost, under the physicalresource constraints and flow conservation constraints. Ourproblem formulation is flexible as it allows one-to-many andmany-to-one NF mapping, and incorporates multipath routingby constructing multiple trees to deliver the multicast service.The formulated problem is an MILP, and thus can be solved toobtain optimal solutions as benchmark. For the multi-servicecase, we present an optimization framework that jointly dealswith multiple service requests. An optimal combination ofservice requests and their joint routing and NF placementconfigurations are studied, such that the aggregate throughput

of the core network is maximized, while the function and linkprovisioning costs are minimized. To reduce the computationalcomplexity in solving the problems in both scenarios, heuristicapproaches are proposed to find accurate solutions close to thatof the optimal solutions. Simulation results are presented todemonstrate the effectiveness and accuracy of the proposedheuristic algorithms.

REFERENCES

[1] O. Alhussein et al., “Joint VNF placement and multicast traffic routingin 5G core networks,” in Proc. IEEE Globecom, Dec. 2018, pp. 1–6.

[2] I. Afolabi, T. Taleb, K. Samdanis, A. Ksentini, and H. Flinck, “Networkslicing and softwarization: A survey on principles, enabling technolo-gies, and solutions,” IEEE Commun. Surveys Tuts., vol. 20, no. 3,pp. 2429–2453, 3rd Quart., 2018.

[3] N. Zhang et al., “Software defined networking enabled wireless networkvirtualization: Challenges and solutions,” IEEE Netw., vol. 31, no. 5,pp. 42–49, May 2017.

[4] H. Huang, P. Li, S. Guo, and W. Zhuang, “Software-defined wirelessmesh networks: Architecture and traffic orchestration,” IEEE Netw.,vol. 29, no. 4, pp. 24–30, Jul. 2015.

[5] S. Zhang, W. Quan, J. Li, W. Shi, P. Yang, and X. Shen, “Air-groundintegrated vehicular network slicing with content pushing and caching,”IEEE J. Sel. Areas Commun., vol. 36, no. 9, pp. 2114–2127, Sep. 2018.

[6] H. Peng, L. Liang, X. Shen, and G. Y. Li, “Vehicular communications:A network layer perspective,” IEEE Trans. Veh. Technol., vol. 68, no. 2,pp. 1064–1078, Aug. 2019.

[7] W. Shi, H. Zhou, J. Li, W. Xu, N. Zhang, and X. Shen, “Drone assistedvehicular networks: Architecture, challenges and opportunities,” IEEENetw., vol. 32, no. 3, pp. 130–137, May 2018.

[8] L. Liu, H. Hu, Y. Luo, and Y. Wen, “When wireless video streamingmeets AI: A deep learning approach,” IEEE Wireless Commun., earlyaccess, Oct. 4, 2019, doi: 10.1109/MWC.001.1900220.

[9] R. Malli, X. Zhang, and C. Qiao, “Benefits of multicasting in all-optical networks,” in Proc. SPIE All-Optical Networking Archit. ControlManag., Oct. 1998, pp. 209–220.

[10] V. V. Vazirani, Approximation Algorithms, 2nd ed. Berlin, Germany:Springer-Verlag, 2004.

[11] E. N. Gilbert and H. O. Pollak, “Steiner minimal trees,” SIAM J. Appl.Math., vol. 16, no. 1, pp. 1–29, 1968.

[12] M. Zhao, B. Jia, M. Wu, H. Yu, and Y. Xu, “Software defined network-enabled multicast for multi-party video conferencing systems,” in Proc.IEEE ICC, Jun. 2014, pp. 1729–1735.

[13] C. Qiao et al., “Virtual network mapping for multicast services Withmax-min fairness of reliability,” J. Opt. Commun. Netw., vol. 7, no. 9,pp. 942–951, 2015.

[14] J. Blendin, J. Ruckert, T. Volk, and D. Hausheer, “Adaptive softwaredefined multicast,” in Proc. IEEE Netsoft, Apr. 2015, pp. 1–9.

[15] S. Q. Zhang, Q. Zhang, H. Bannazadeh, and A. Leon-Garcia, “Rout-ing algorithms for network function virtualization enabled multicasttopology on SDN,” IEEE Trans. Netw. Service Manag., vol. 12, no. 4,pp. 580–594, Dec. 2015.

[16] S. Q. Zhang, A. Tizghadam, B. Park, H. Bannazadeh, andA. Leon-Garcia, “Joint NFV placement and routing for multicast serviceon SDN,” in Proc. IEEE/IFIP NOMS, Apr. 2016, pp. 333–341.

[17] M. Zeng, W. Fang, and Z. Zhu, “Orchestrating tree-type VNF forwardinggraphs in inter-DC elastic optical networks,” J. Lightw. Technol., vol. 34,no. 14, pp. 3330–3341, Jul. 15, 2016.

[18] Z. Xu, W. Liang, M. Huang, M. Jia, S. Guo, and A. Galis, “Approxi-mation and Online algorithms for NFV-enabled multicasting in SDNs,”in Proc. IEEE ICDCS, Jun. 2017, pp. 625–634.

[19] J.-J. Kuo, S.-H. Shen, M.-H. Yang, D.-N. Yang, M.-J. Tsai, andW.-T. Chen, “Service overlay forest embedding for software-definedcloud networks,” in Proc. IEEE ICDCS, Jun. 2017, pp. 720–730.

[20] L. Huang, H. Hung, C. Lin, and D. Yang, “Scalable steiner treefor multicast communications in software-defined networking,” CoRR,vol. abs/1404.3454, May 2014.

[21] S. Mehraghdam, M. Keller, and H. Karl, “Specifying and placing chainsof virtual network functions,” in Proc. IEEE CloudNet, Oct. 2014,pp. 7–13.

[22] M. F. Bari, S. R. Chowdhury, R. Ahmed, and R. Boutaba, “Onorchestrating virtual network functions,” in Proc. CNSM, Nov. 2015,pp. 50–56.


http://dx.doi.org/10.1109/MWC.001.1900220


[23] M. C. Luizelli, L. R. Bays, L. S. Buriol, M. P. Barcellos, andL. P. Gaspary, “Piecing together the NFV provisioning puzzle: Effi-cient placement and chaining of virtual network functions,” in Proc.IFIP/IEEE IM, May 2015, pp. 98–106.

[24] M. Ghaznavi, N. Shahriar, S. Kamali, R. Ahmed, and R. Boutaba,“Distributed service function chaining,” IEEE J. Sel. Areas Commun.,vol. 35, no. 11, pp. 2479–2489, Nov. 2017.

[25] Y. Xie, Z. Liu, S. Wang, and Y. Wang, “Service function chainingresource allocation: A survey,” CoRR, vol. abs/1608.00095, Aug. 2016.

[26] S. Q. Zhang et al., “Sector: TCAM space aware routing on SDN,” inProc. IEEE ITC, Sep. 2016, pp. 216–224.

[27] J. Li, W. Shi, Q. Ye, W. Zhuang, X. Shen, and X. Li, “Online jointVNF chain composition and embedding for 5G networks,” in Proc. IEEEGlobecom, Dec. 2018, pp. 1–6.

[28] Q. Ye, J. Li, K. Qu, W. Zhuang, X. S. Shen, and X. Li, “End-to-endquality of service in 5G networks: Examining the effectiveness of anetwork slicing framework,” IEEE Veh. Technol. Mag., vol. 13, no. 2,pp. 65–74, Jun. 2018.

[29] W. Wang, B. Liang, and B. Li, “Multi-resource generalized processorsharing for packet processing,” in Proc. IEEE/ACM IWQoS, Jun. 2013,pp. 1–10.

[30] A. Ghodsi, V. Sekar, M. Zaharia, and I. Stoica, “Multi-resource fairqueueing for packet processing,” ACM SIGCOMM Comput. Commun.Rev., vol. 42, no. 4, pp. 1–12, Sep. 2012.

[31] A. Ghodsi, M. Zaharia, B. Hindman, A. Konwinski, S. Shenker,and I. Stoica, “Dominant resource fairness: Fair allocation of multipleresource types,” in Proc. USENIX NSDI, 2011, pp. 323–336.

[32] M. Chowdhury, M. R. Rahman, and R. Boutaba, “ViNEYard: Virtual net-work embedding algorithms with coordinated node and link mapping,”IEEE/ACM Trans. Netw., vol. 20, no. 1, pp. 206–219, Feb. 2012.

[33] M. R. Garey and D. S. Johnson, Computers intractability: A guide totheory NP-completeness, 1st ed. San Francisco, CA, USA: Freeman,1979.

[34] S. Chopra, “Comparison of formulations and a heuristic for packingsteiner trees in a graph,” Ann. Oper. Res., vol. 50, no. 1, pp. 143–171,Dec. 1994.

[35] A. M. Abbas and B. N. Jain, “Mitigating path diminution in disjointmultipath routing for mobile ad hoc networks,” Int. J. Ad Hoc UbiquitousComput., vol. 1, no. 3, p. 137, 2006.

[36] A. M. Abbas and B. N. Jain, “Path diminution in node-disjoint multipathrouting for mobile ad hoc networks is unavoidable with single routediscovery,” Int. J. Ad Hoc Ubiquitous Comput., vol. 5, no. 1, p. 7, 2010.

[37] Q. Ye, W. Zhuang, X. Li, and J. Rao, “End-to-end delay modeling forembedded VNF chains in 5G core networks,” IEEE Internet ThingsJ., vol. 6, no. 1, pp. 692–704, Feb. 2019.

[38] K. R. Anderson, “A reevaluation of an efficient algorithm for determin-ing the convex hull of a finite planar set,” Inf. Process. Lett., vol. 7,no. 1, pp. 53–55, Jan. 1978.

[39] N. Zhang, Y.-F. Liu, H. Farmanbar, T.-H. Chang, M. Hong, andZ.-Q. Luo, “Network slicing for service-oriented networks underresource constraints,” IEEE J. Sel. Areas Commun., vol. 35, no. 11,pp. 2512–2521, Nov. 2017.

[40] M. T. Beck and J. F. Botero, “Coordinated allocation of service functionchains,” in Proc. IEEE Globecom, Dec. 2014, pp. 1–6.

Omar Alhussein (Student Member, IEEE) receivedthe B.Sc. degree in communications engineeringfrom Khalifa University, Abu Dhabi, United ArabEmirates, in 2013, and the M.A.Sc. degree in engi-neering science from Simon Fraser University, Burn-aby, BC, Canada, in 2015. He is currently pursuingthe Ph.D. degree in electrical engineering with theUniversity of Waterloo, Waterloo, ON, Canada. Hisresearch interests include next-generation wirelessnetworks, network (function) virtualization, wirelesscommunications, and machine learning.

Phu Thinh Do (Member, IEEE) received the B.S.degree in electrical engineering from the Ho ChiMinh City University of Technology, in 2010, andthe M.S.E. and Ph.D. degrees from the Departmentof Electronics and Radio Engineering, Kyung HeeUniversity, South Korea, in 2016. From 2016 to2019, he was first a Post-Doctoral Fellow with theDigital Communication Lab, Kyung Hee University,then with the Broadband Communications ResearchLab, University of Waterloo, Canada. He is currentlya Researcher with the Institute of Research and

Development, Duy Tan University, Vietnam. His current research interestsinclude 5G wireless communications, search engines, and recommendationsystems for e-commerce platforms.

Qiang Ye (Member, IEEE) received the Ph.D.degree in electrical and computer engineering fromthe University of Waterloo, Waterloo, ON, Canada,in 2016. He was a Post-Doctoral Fellow with theDepartment of Electrical and Computer Engineering,University of Waterloo, from December 2016 toNovember 2018, where he was a Research Associatefrom December 2018 to September 2019. He hasbeen an Assistant Professor with the Departmentof Electrical and Computer Engineering and Tech-nology, Minnesota State University, Mankato, MN,

USA, since September 2019. His current research interests include 5Gnetworks, software-defined networking and network function virtualization,network slicing, artificial intelligence and machine learning for future net-working, protocol design, and end-to-end performance analysis for the Internetof Things.

Junling Li (Student Member, IEEE) received theB.S. degree from Tianjin University, Tianjin, China,in 2013, and the M.S. degree from the Beijing Uni-versity of Posts and Telecommunications, Beijing,China, in 2016. She is currently pursuing the Ph.D.degree with the Department of Electrical and Com-puter Engineering, University of Waterloo, Waterloo,ON, Canada. Her interests include SDN, NFV, net-work slicing for 5G networks, machine learning,and vehicular networks. She received the Best PaperAward at the IEEE/CIC International Conference onCommunications in China (ICCC) in 2019.

Weisen Shi (Student Member, IEEE) received theB.S. degree from Tianjin University, Tianjin, China,in 2013, and the M.S. degree from the Beijing Uni-versity of Posts and Telecommunications, Beijing,China, in 2016. He is currently pursuing the Ph.D.degree with the Department of Electrical and Com-puter Engineering, University of Waterloo, Waterloo,ON, Canada. His interests include drone communi-cation and networking, network function virtualiza-tion, and vehicular networks.



Weihua Zhuang (Fellow, IEEE) has been with theDepartment of Electrical and Computer Engineer-ing, University of Waterloo, Canada, since 1993,where she is currently a Professor and the Tier ICanada Research Chair in wireless communicationnetworks. She was a recipient of the 2017 TechnicalRecognition Award from the IEEE CommunicationsSociety Ad Hoc and Sensor Networks TechnicalCommittee, and several best paper awards fromIEEE conferences. She is a Fellow of the RoyalSociety of Canada, the Canadian Academy of Engi-

neering, and the Engineering Institute of Canada. She is an Elected Memberof the Board of Governors and VP Publications of the IEEE VehicularTechnology Society. She was the Technical Program Chair/Co-Chair ofthe IEEE VTC 2016 Fall and 2017 Fall, the Editor-in-Chief of the IEEETRANSACTIONS ON VEHICULAR TECHNOLOGY from 2007 to 2013, and theIEEE Communications Society Distinguished Lecturer from 2008 to 2011.

Xuemin (Sherman) Shen (Fellow, IEEE) receivedthe Ph.D. degree in electrical engineering fromRutgers University, New Brunswick, NJ, USA, in1990.

He is currently a University Professor with theDepartment of Electrical and Computer Engineer-ing, University of Waterloo, Canada. His researchfocuses on network resource management, wirelessnetwork security, social networks, 5G and beyond,and vehicular ad hoc and sensor networks. He isalso a registered Professional Engineer of Ontario,

Canada, an Engineering Institute of Canada Fellow, a Canadian Academy ofEngineering Fellow, a Royal Society of Canada Fellow, a Chinese Academyof Engineering Foreign Fellow, and a Distinguished Lecturer of the IEEEVehicular Technology Society and Communications Society. He received theR. A. Fessenden Award in 2019 from IEEE, Canada, James Evans Avant GardeAward in 2018 from the IEEE Vehicular Technology Society, Joseph LoCiceroAward in 2015 and Education Award in 2017 from the IEEE CommunicationsSociety. He has also received the Excellent Graduate Supervision Award in2006 and Outstanding Performance Award five times from the Universityof Waterloo and the Premier’s Research Excellence Award (PREA) in 2003from the Province of Ontario, Canada. He has served as the Technical ProgramCommittee Chair/Co-Chair for the IEEE Globecom’16, the IEEE Infocom’14,the IEEE VTC’10 Fall, the IEEE Globecom’07, the Symposia Chair forthe IEEE ICC’10, the Tutorial Chair for the IEEE VTC’11 Spring, andthe Chair for the IEEE Communications Society Technical Committee onWireless Communications. He was the Editor-in-Chief of the IEEE INTERNETOF THINGS JOURNAL and the Vice President on Publications of the IEEECommunications Society.

Xu Li (Member, IEEE) received the B.Sc. degreefrom Jilin University, China, in 1998, the M.Sc.degree from the University of Ottawa in 2005, andthe Ph.D. degree from Carleton University in 2008,all in computer science. He worked as a ResearchScientist (with tenure) with Inria, France. He iscurrently a Senior Principal Researcher with HuaweiTechnologies Canada. His current research interestsare focused in 5G and beyond. He has publishedmore than 100 refereed scientific articles. He isholding more than 40 issued U.S. patents. He has

contributed extensively to the development of 3GPP 5G standards throughmore than 90 standard proposals.

Jaya Rao (Member, IEEE) received the B.S. andM.S. degrees in electrical engineering from theUniversity at Buffalo, Buffalo, NY, USA, in 2001and 2004, respectively, and the Ph.D. degree fromthe University of Calgary, Canada, in 2014. He iscurrently a Senior Research Engineer with HuaweiTechnologies Canada, Ottawa. Since joining Huaweiin 2014, he has worked on research and designof URLLC, V2X, and Industrial IoT (IIoT)-basedsolutions in 5G New Radio (NR). He has partici-pated in 5G system impact analysis and performance

assessments for contributing towards commercial alliance bodies, such asNGMN. He has also contributed at 3GPP RAN WG2, RAN WG3, and SA2meetings (Rel-15 & Rel-16) on topics related to URLLC, NR V2X, IIoT,Network Slicing, Mobility Management, and Session Management. From2004 to 2010, he was a Senior Research Engineer with Motorola Inc. He wasa recipient of the Best Paper Award at the IEEE WCNC 2014.


Documents

A Virtual Network Customization Framework for Multicast