Embed Size (px)
Citation preview
Real time alpha-fairness based traffic engineering
Bill McCormickHuawei Technologies Canada
Kanata Ontario, [email protected]
Frank KellyUniversity of Cambridge
Cambridge, [email protected]
Patrice PlanteHuawei Technologies Canada
Kanata, Ontario, [email protected]
Paul GunningBT Research & InnovationAdastral Park, Ipswich, [email protected]
Peter Ashwood-SmithHuawei Technologies Canada
Kanata, Ontario, [email protected]
ABSTRACTSoftware defined networking (SDN) traffic engineering hasproved to be a computationally difficult problem, resultingin long execution times to compute large scale flow assign-ments. Other authors have taken approximate approaches toachieving fairness to improve computational scalability. Wederive a solution to the general alpha-fairness problem thatscales near-linearly with the problem size and is well suitedto a massively parallel implementation. CPU based resultscompare scalability and accuracy with proportional fair andmax min fair solutions on a simulation of BT’s productionnetwork. Results from our FPGA implementation show athree order of magnitude reduction in execution time, allow-ing large scale flow assignment computations to take placein times measured in milliseconds instead of seconds or min-utes. We believe that these techniques will finally enable realtime traffic engineering in SDN.
1. INTRODUCTIONOne of the objectives of SDN is to remove the more
complex functions from routers and virtualize them ina data center. In today’s networks, routers implementshortest path based routing protocols which are usedto route traffic through the network and provide eitherde jure traffic engineering (as in MPLS-TE) or de factotraffic engineering (as in OSPF). When these shortestpath first (SPF) routing functions are no longer presentin the router, a new traffic engineering function is re-quired. Instead of being distributed through the net-work, this function can be centralised. This centralfunction can have global knowledge of the network thatis not available to the distributed SPF routing in to-day’s network.
Two key network attributes related to traffic engi-neering are network throughput and fairness. (Anotherkey attribute is delay, however we do not address delaydirectly in this paper.) We balance these attributes us-ing the concept of alpha fairness introduced by Mo andWalrand in [11] where α is a parameter in the range
[0,∞] to denote fairness.There are three specific values of α which are of in-
terest. Setting α = 0 corresponds to a flow assignmentwhich maximizes the network throughput, but makesno attempt to ensure fairness among flow assignments.
As α → ∞, the flow assignment becomes max-minfair. A flow assignment is max-min fair when the band-width assigned to a flow can only be increased by de-creasing the bandwidth assigned to some other flowwith an equal or smaller assignment. Thus max-minfairness is focused on making the minimum flow assign-ment as large as possible without regard to the impacton total throughput.
Setting α = 1 corresponds to a proportional fair so-lution. Proportional fair solutions were first describedby Nash [12] as the solution to a negotiation problem.They provide an appealing compromise between max-min fairness - which allocates flows fairly without regardfor network resource usage - and maximal throughput -which allocates flows without regard for fairness.
Optimization programs to solve these flow assignmentproblems are given in [13]. The maximum throughputproblem can be solved with a single linear program. Theproportional fair problem requires a convex program, soa traditional linear solver will not suffice here. The max-min fair problem requires the solution of a sequenceof linear programs which grows polynomially with theproblem size. Techniques for solving these problemsall exhibit polynomial computation scalability, as tra-ditional solutions require the repeated factoring of amatrix which grows with the problem size.
Recent papers [9] and [10] describe practical trafficengineering implementations on data center networkswith less than one hundred nodes. In [9], the authorsrelax the max-min fairness constraints to reduce thenumber of linear programs needed to solve the optimiza-tion problem. In [10], the authors describe a greedy al-gorithm which approximates max-min fairness withoutthe cost of executing a sequence of linear programs.
The focus of our work is on large carrier networks,
1
ranging in size from one hundred to a few thousandnodes. We present a new method for solving theseproblems which scales near-linearly with the problemsize and is also well suited to a massively parallel im-plementation.
2. ALGORITHM DERIVATIONModel the network as a set of J directed links, indi-
vidually identified as j ∈ J . Each link has capacity Cj .The term r is used to identify a specific path throughthe network. An individual flow is identified by the terms. The bandwidth assigned to a specific flow is identi-fied by xs, and the bandwidth from flow s assigned topath r is identified by yr. We use the terminology r ∈ sto denote the paths that are used by a specific flow andr ∈ j to denote the paths that use link j. When we arereferring to a specific path r, we use the expression s(r)to denote the parent flow of the path.
The optimization program we use for a weighted αfair flow assignment is given by
maximize∑s∈S w
αsx1−αs
1−αsubject to
∑r∈s yr = xs,∑r∈j yr ≤ Cj
over x, y > 0
The term ws is a weight assigned to each flow, allowingthe user to request that some flows be assigned propor-tionally more or less bandwidth than others.
This program has unique values for x, however thesolution for y is usually non-unique. If we define
xs =
(∑r∈s
yqr
) 1q
where q is some constant close to, but less than, one,then the optimization problem has a unique solution forboth the x values and the y values. With this change,our objective function becomes the convex function
maximize∑s∈S
wαs
(∑r∈s y
qr
) 1−αq
1− α
Returning to first principles, we can construct theLagrangian for this problem as
L(y, z;µ) =∑s w
αs
(∑r∈s yr
q)1−αq
1−α +∑j µj
(Cj −
∑r∈j yr − zj
)Here zj and µj are slack variables and shadow pricesfor link j respectively. From complementary slackness,we know that for a given j, either µj = 0 or zj = 0[6]. In other words, at the solution to our optimizationproblem, if the shadow price is non-zero then link j
is saturated, and if link j is under committed then itsshadow price is 0.
We can differentiate L with respect to yr to developa relationship between y, x and µ.
∂L
∂yr= wαs(r)y
q−1r
∑r′∈s(r)
yqr′
1−αq −1
−∑j∈r
µj .
At the optimum point this derivative will be equal tozero. Setting ∂L
∂yr= 0 and rearranging we find that
yr =
((ws(r)
xs(r)
)α· 1∑
j∈r µj
) 11−q
xs(r)
Update rules for Xs and µj should be of the form
µj(t+ 1) = µj(t) + kjµj(t)∆t
xs(t+ 1) = xs(t) + ksxs(t)∆t
where kj and ks are gain parameters for the update rulesfor µ and x respectively, and the dot notation denotesthe time derivative.
In [14], Thomas Voice shows that this class of sys-tem converges to the optimal value provided that thegain parameters are constrained appropriately. Follow-ing [14] and setting the gain parameters to their maxi-mum stable values gives the optimization algorithm as
yr =
((ws(r)
xs(r)
)α· 1∑
j∈r µj
) 11−q
xs(r) (1)
µj(t+ 1) = µj(t) +1− q
2µj(t)
[∑r∈j yr(t)− Cj
Cj
](2)
xs(t+ 1) = xs(t) +1− q
2(α+ q − 1)xs(t) ·[∑
r∈s yr(t)q − xs(t)q
xs(t)q
](3)
Note carefully that each of the update rules in equa-tions (1), (2) and (3) can be implemented in parallel. Inother words, all of the yr values in (1) can be computedin parallel, then all of the µj values in (2) can be com-puted and so on. This property makes it straightfor-ward to implement the algorithm on massively parallelhardware.
3. SIMULATION RESULTSIn order to assess the performance of this algorithm,
we have run simulations comparing the algorithm re-sults to reference implementations for max-min fairnessand proportional fairness. Our Lagrangian based al-gorithm is implemented in Java 7. For the referenceimplementations we use general purpose open sourcesolvers written in C and FORTRAN as detailed below.
2
Figure 1: BT 21CN network
The simulations are run on an x86 based virtual ma-chine. We aren’t concerned with making absolute per-formance comparisons at this time - the purpose of thisstep is to assess accuracy and to obtain a general feelfor the computational complexity of the algorithm ascompared to the traditional implementations.
We have used BT’s 21CN network topology (Figure1) comprising 106 nodes and 234 links within the UnitedKingdom as a reference for these simulations. Flows aregenerated using a pseudo-random number generator sothat the end points for each flow are randomly selected.All flows are treated as elastic, so they will consume allnetwork bandwidth available to them.
3.1 Max-min FairnessOur max-min fairness reference implementation uses
the GNU linearing programming kit [1]. This is a scal-able open source linear solver written in C. The ref-erence algorithm is Algorithm 8.3 from [13]. For ourLagrangian algorithm, we choose q = 0.9 and α = 4 asan approximation for max-min fairness.
The simulation results are shown in Figures 2 and3. As expected, the execution time grows rapidly withthe problem size for the reference algorithm as largerproblems require execution of a growing number of lin-ear programs. Our algorithm shows a roughly linearincrease in execution time with problem size. Choice ofq = 0.9 provides a good approximation of max-min fair,holding the root mean square error from the referenceimplementation at around 1%.
0
5
10
15
20
25
0 100 200 300
Ex
ecu
tio
n t
ime
(se
con
ds)
Number of flows
GLPK Lagrangian
Figure 2: Max-min fair execution time
0
0.01
0.02
0 100 200 300
Re
lati
ve
RM
S e
rro
r
Number of flows
Figure 3: Max-min fair RMS error
3.2 Proportional FairnessOur proportional fairness reference implementation
requires a convex optimizer as it has a non-linear ob-jective function. For this simulation we have used theinterior point optimizer Ipopt [3], an open source libraryknown for its good scalability properties. This libraryis written in C and FORTRAN and we have configuredit with the MUMPS linear solver [4]. The referencealgorithm here is from section 8.1.3 of [13].
The proportional fair simulation results are shown inFigures 4 and 5. In this case, the reference implemen-tation requires the execution of a single non-linear opti-mization program, so it doesn’t exhibit the higher orderpolynomial growth of the max-min fair implementation.Our Lagrangian based method generally matches theperformance of the reference implementation. As withthe max-min fair example, choice of q = 0.9 keeps theRMS error to approximately 0.5%.
4. FPGA IMPLEMENTATIONWe have implemented our Lagrangian based algo-
rithm in a Xilinx FPGA Virtex-7 XC7VX485T-3. The
3
0
1
2
3
4
5
6
7
0 500 1000 1500
Exe
cuti
on
tim
e (
seco
nd
s)
Number of flows
Ipopt
Lagrangian
Figure 4: Proportional fair execution time
0
0.01
0.02
0.03
0 500 1000 1500
Re
lati
ve R
MS
err
or
Number of flows
Figure 5: Proportional fair RMS error
goal of the hardware implementation is to take advan-tage of the parallelism that can be found in the algo-rithm design to evaluate how we can improve the con-vergence time by executing concurrently as many oper-ations as possible.
As illustrated in the block diagram of Figure 7, thehardware implementation consists of 1024 parallel addersthat feed 32 parallel Deep Pipelines. Pipelining is apowerful concept that allows multiple operations to beexecuted sequentially and simultaneously on large inde-pendent data samples.
In our implementation, each Deep Pipeline is madeof 180 pipeline stages consisting of 12 Divide, 15 Multi-ply, 4 Square Root, 44 Add and 2 Subtract operations.When a Deep Pipeline is fully populated, that singlepipeline can produce one partial result of equation (1),(2) and (3) every clock cycle. With a core clock run-ning at 200MHz, the FPGA produces 500G fixed pointoperations per second.
Performance results for our FPGA implementationare shown in Figure 6. This figure shows flow compu-tation results for flow assignment with q = 0.75 andα = 4.0 to provide a reasonable approximation of max-
0
0.5
1
1.5
2
2.5
3
0 5000 10000
Exe
cuti
on
tim
e (
mse
c)
Number of flows
Figure 6: FPGA max-min fair execution time
min fair flow assignment. The notable difference be-tween these results and those in Figure 2 are that theexecution time has dropped from seconds to millisec-onds.
Comparing the two results, we attribute the 3 or-der of magnitude improvement to hardware pipelining.In order to benefit from hardware pipelining, the net-work size must be large enough to keep the pipeline fullmost of the time. This effect is illustrated in Figure6, where performance reaches a steady state for greaterthan 1000 active flows by avoiding the penalty cost offilling and emptying a partially filled pipeline.
5. DISCUSSION
5.1 PerformanceWhen we began work on this project, it was clear
that solving flow assignment problems is a compute in-tensive task. Modern processors have lots of processingpower, but its available in a highly parallel form. Forexample, a NVIDIA K20 has 3.52 TFLOPs spread over2496 cores [5]. An Intel CoreTM i7-980 has 80 GFLOPsavailable over 6 cores [2]. The compute throughput ofa field programmable gate array is not specified in thismanner, however in [7], the authors demonstrated 180GFLOPS of practical compute throughput on a Virtex-7 FPGA.
In order to make effective use of these compute en-vironments, it was necessary that the algorithm designmap directly to a highly parallel architecture. We feelthat the dramatic performance gains of our hardwareimplementation validates our approach.
5.2 ConvergenceA significant factor in execution time is the number
of iterations required to converge. We detect conver-gence by measuring the relative change in the norm of
4
Fixed Point Operation Pipeline
180 Pipeline Stages x32bit 200MHz
Configuration
Table
20 Mbit
QPI
PHY
Path Table
6 Mbit
Flow Table
1 Mbit
Link Table
1 Mbit
CPU
Interface
FPGA
1024
AddersCombiner
Deep Pipeline #0
Deep Pipeline #1
Deep Pipeline #30
Deep Pipeline #31
...
Figure 7: FPGA block diagram
the vector of yr values. When this change drops belowa threshold (10−6 in our examples), the algorithm isstopped.
Empirically the number of iterations to convergencehas varied in the range [200, 2400]. There appears tobe a direct relationship between the iterations to con-verge and the number of link constraints that are activeor almost active. As the number of active constraintsincreases, the algorithm takes more time to explore theproblem structure and converge to a solution. This is atopic for further study.
5.3 Further FPGA design considerationsThe hardware implementation is limited by the sil-
icon area of the FPGA. The algorithm could be fur-ther accelerated by adding even more Deep Pipelinesin parallel in a larger FPGA device. To reduce sili-con consumption by the Deep Pipeline, our hardwareimplementation uses fixed point operations instead offloating point operations. To avoid floating point op-erations, the Deep Pipeline is divided in two distinctregions. The two regions use different fixed point repre-sentation and are merged by a multiplication operation.
The bulk of the FPGA real estate is consumed inmaintaining the configuration table that holds the net-work definition. This information could be stored inexternal RAM which would allow us to increase the sizeof the Link, Flow and Path tables which are currentlyconfigured to support 512 links, 32K flows, 96K pathsand 12 links per path.
The latest FPGA offerings support a core clock fre-quency of 400Mhz, while our implementation uses a200Mhz core clock. Migrating to one of these FPGAswould double the computational performance.
6. CONCLUSIONOur FPGA implementation is by far the fastest im-
plementation of the max-min fair and the more gen-eral α-fairness problem we have seen published. It ex-hibits near-linear computational and memory scalabil-ity which makes it an excellent choice for use on largecarrier scale networks.
The algorithm has desirable implementation proper-ties in terms of simplicity. Instead of the complicatedmatrix factorization used in interior point methods [3][4] [6], we use straightforward vector operations for up-date rules. The implementation team does not need tobe versed in arcane rules for matrix computations andcan focus on the more practical problems of identifyingperformance bottlenecks.
Our solution runs in millisecond times, making it fastenough to be part of the real time control loop in thenetwork instead of an off-line tool. This means that wefinally have line of sight into real time traffic engineeringas originally envisaged for Software Defined Networking[8].
7. REFERENCES[1] Gnu linear programming kit.
http://www.gnu.org/software/glpk. Accessed:
5
2014-03-06.[2] Intel microprocessor export compliance metrics.
http://www.intel.com/support/processors/sb/CS-032814.htm. Accessed:2014-03-20.
[3] Interior point optimizer.http://project.coin-or.org/Ipopt. Accessed:2014-03-06.
[4] Mumps: a multifrontal massively parallel sparsedirect solver. http://mumps.enseeiht.fr. Accessed:2014-03-06.
[5] Tesla Kepler GPU accelerators.http://www.nvidia.com/content/tesla/pdf/Tesla-KSeries-Overview-LR.pdf. Accessed:2014-03-06.
[6] S. Boyd and L. Vandenberghe. ConvexOptimization. Cambridge University Press, NewYork, 2004.
[7] J. Capello and D. Strenski. A practical measure ofFPGA floating point acceleration for highperformance computing. In 2013 IEEE 24thInternational Conference on Application-SpecificSystems, Architectures and Processors, pages160–167, 2013.
[8] S. Das, N. McKeown, et al. Application-awareaggregation and traffic engineering in a covergedpacket-circuit network. In Proceedings ofOFC/NFOEC 2011, pages 1–3, 2011.
[9] C.-Y. Hong et al. Achieving high utilization withsoftware-driven wan. In Proceedings of the ACMSIGCOMM 2013, pages 15–26, 2013.
[10] S. Jain et al. B4: Experience with aglobally-deployed software defined wan. InProceedings of the ACM SIGCOMM 2013, pages3–14, 2013.
[11] J. Mo and J. Walrand. Fair end-to-endwindow-based congestion control. IEEE/ACMTransactions on Networking, 8(5):556–567,October 2000.
[12] J. Nash. The bargaining problem. Econometrica,18(2):155–162, April 1950.
[13] M. Pioro and D. Medhi. Routing, Flow, andCapacity Design in Communication andComputer Networks. Morgan KaufmannPublishers, 2004.
[14] T. Voice. Stability of multi-path dual congestioncontrol algorithms. IEEE/ACM Transactions onNetworking, 15(6):1231–1239, December 2007.
6
