Constant Time Weighted Frequency Estimation forVirtual Network Functionalities

Gil EinzigerNokia Bell Labs

gil.einziger@nokia.com

Marcelo Caggiani LuizelliFederal University of Rio Grande do Sul

(UFRGS) Brazilmcluizelli@inf.ufrgs.br

Erez WaisbardNokia Bell Labs

erez.waisbard@nokia.com

Abstract—Monitoring flow volumes is a fundamental capabilityin network measurement. Sampling is often used to cope withthe line speed and the applied methods typically rely on uniformpacket sampling. However, it is inaccurate when there is a largevariance in packet sizes.

In this work we introduce Byte Uniform Sampling (BUS), asampling method for estimating flow volumes. We show thatBUS can be combined with existing unweighted estimationalgorithms and that the result is a weighted algorithm. BUSenables an asymptotic update time improvement as existingweighted algorithms are slower. We formally analyze BUS andevaluate it on five Internet traces. Finally, we extend the DPDKversion of Open vSwitch to support BUS and demonstrate similarthroughput when compared to uniform packet samples.

I. INTRODUCTION

Network functionalities such as traffic engineering, loadbalancing, quality of service, caching and anomaly/intrusiondetection [1]–[6] often rely on on-line per-flow measurements.Such measurements can focus on packet and byte counts andthe desired metric differs from one application to another. Forexample, load balancers deal with the physical limitations oflinks and therefore require byte volume information.

In practice, performing measurement at line speed is adifficult challenge due to the large number of flows and therapid transmission rates [7]–[11]. Sampling is a fundamentalmeasurement technique as it only processes a small fraction ofthe traffic. Indeed, applied technologies such as sFlow [12] andNetFlow [13] offer sampling capabilities. Unfortunately, theyonly support random samples that are inefficient for estimatingbyte volumes. The problem escalates when packets’ sizevariance is large. Specifically, in some real Internet traces [14],[15] less than 1% of the packets constitute up to 25% ofthe overall byte traffic. Clearly, a load balancer that relies onpacket counts may not be able to correctly balance the trafficdue to inaccurate sample input.

Another key challenge of on-line measurements is the mas-sive data scale. Modern network devices handle several millionflows [16] at the same time, and as the measurement progressesthe number of monitored flows increases. Algorithms foridentifying the most frequent flows, which are sometimescalled heavy hitters, reduce the required space [17]–[21].

The need for software-based on-line measurement is alsoat the core of emerging network paradigms such as Network

Fig. 1: An example of the contribution in this paper. BUSsamples packets with probability proportional their byte sizeand sampled packets are counted with a legacy measurementalgorithm. The result is a weighted meta algorithm that insome cases asymptotically improves the state of the art.

Function Virtualization (NFV). NFV allows network functio-nalities to be executed in a virtual manner on commodity ser-vers. That is, it reduces dependence on proprietary hardware,increases flexibility and lowers operation costs.

This work introduces a new sampling method called ByteUniform Sampling (BUS) and combines it with existing stre-aming algorithms for finding frequent items. Intuitively, sam-ples provide the required speed, while the streaming algorithmhandles the space limitations. We show that the combinedmethod supports weights and can monitor flow’s byte volumeby counting the number of its sampled packets. This enablesan asymptotic update time improvement by using faster algo-rithms that can only count packets. Our approach is illustratedin Figure 1.

A. Contributions

We introduce a new sampling method called Byte UniformSampling (BUS). We prove that BUS provides accuracy gua-rantees and unbiased volume estimations. We then explain howBUS can be combined with existing streaming algorithms andstill provide strong accuracy guarantees. BUS is extensivelyevaluated using five Internet packet traces and we demonstrateits accuracy improvement over uniform packet samples. Thedegree of improvement depends on the packets’ size variance.In some real traffic traces, uniform packet sampling requiresover 50 times as many samples to match the accuracy of BUS(see Figure 3a).

We show that BUS can be combined with legacy streamingalgorithms to create a faster algorithm that supports weights.When the underlying algorithm is Space Saving [20], weasymptotically improve the update complexity for weightedinputs, from logarithmic to constant. The reason for that isthat Space Saving implementations operate in constant time forunweighted inputs and we can use that implementation for bytetraffic measurement. We analyze the accuracy of this mode ofoperation and show that it provides strong accuracy guarantees.We also evaluate it and demonstrate similar accuracy to theunderlying Space Saving algorithm, even when sampling lessthan 1% of the traffic.

Finally, we implement BUS in a DPDK-enabled OpenvSwitch (OVS) and evaluate its overheads on generated traffic.We show that BUS achieves similar throughput to commonlyused uniform packet samples.

II. RELATED WORK

A. Streaming algorithms

Streaming algorithms reduce the space required for perfor-ming measurement at the expense of precision. That is, theyprovide approximate measurements in a tractable space. Thesealgorithms typically process every packet in the stream andrequire logarithmic complexity to measure per-flow volume.Alternatively, some algorithms [20], [21] can count packetsin constant time. Logarithmic runtime, as well as the need toprocess every packet in the stream, makes them insufficientfor on-line measurements.

Streaming algorithms come with two distinct flavors, sketchand counter based algorithms. Sketch based methods arecomposed of counter arrays, that are updated by multiple hashfunctions. Classical examples include Count Sketch (CS) [19]and Count Min Sketch (CMS) [18]. Sketches are simple toimplement and are widely used in practice. While sketches areconsidered fast, increasing line speeds has motivated faster andmore accurate off-line sketches. For example, Counter Braidsand Counter Tree [10], [11], improve accuracy with off-linedecode algorithms that reconstruct the counter values. Rando-mized Counter Sharing [22] updates a single counter and usesMaximum Likelihood Estimation to reconstruct frequencies.

Counter algorithms maintain a small flow cache and attemptto populate their cache with the largest or most frequent flows.They differ from each other in the cache’s maintenance policy.Examples include Lossy Counting [23] and its extensions [24],[25], Frequent [21] and Space Saving (SS) [17], [20]. Speci-fically, when a packet of an unmonitored flow arrives, SpaceSaving admits the newly arriving flow at the expense of thesmallest monitored flow. Additionally, when a new flow isadmitted to the cache, its estimated frequency starts from thatof the evicted flow. Space Saving is often considered the stateof the art [26]–[28].

B. Sampling techniques

Samples are a prominent approach for network measure-ment as they require to process only a small fraction of thetraffic. The most common method is Packet Uniform Sampling

(PUS) and is often the one used in practice [13], [12]. Whileconceptually simple, PUS can be ineffective for approximatingbyte counts since every packet is sampled with the sameprobability. As a result, large packets have a smaller perbyte sample probability. The accuracy of PUS depends onthe variance of the packets’ byte size and [29] dynamicallyadjust the sample probability to compensate for that variance.That is, when the variance is larger than normal, the sampleprobability is increased to compensate. Similar techniques areused by [30], [31] that also support constraints on resour-ces. Similarly, [32] develops a sampled based billing servicethat identifies users that exceeded a certain quota withoutmonitoring every user. Sampling can be used to measurediverse statistical features [33], and [34] even partition asampling budget to satisfy multi-objective goals. In hardware,estimators [16], [35], [36] use samples to reduce the size ofcounters and fit more of them in expensive SRAM memory.

III. BYTE UNIFORM SAMPLING (BUS)

A. Preliminaries

Our measurement is modeled by a sequence of packets (S),which is initially empty and at each step an additional packetis added to (S). Packets are modeled as id and weight tuples(< id,w >). Where id ∈ U , w ∈ N \ {0} and U is thedomain of ids. Packets that share the same id belong to thesame flow and our goal is to estimate the total weight of aflow (Bid). M and A are the maximal and average packetsize, M is assumed to be known in advance. N and B are thetotal number of packets and byte in S. Table I summarizes thenotations used in this work.

Symbol Meaning

S Stream

N Current number of packets (in all flows)

B Current number of bytes (in all flows)

M Maximal packet weight

A Average packet weight

Six Random variable, Six = 1 if x’s i’th packet is sampled.

Sx the number of sampled packets from flow x.

S Total number of sampled packets from all flows.

wi Weight of the i’th packet in bytes.

wix Weight of the i’th packet in flow x (bytes).

U Domain of ids.

pi i’th packet in the stream. pi =< di, wi >.

Px Total number of packets in flow x

Px An estimation of PxBx The total number of bytes in flow x

Bx An estimation of Bxεs, εa, ε Accuracy parameters

δs, δ Confidence parameters

τ Average packet sampling probability

β Average byte sampling probability

TABLE I: List of symbols

Given a flow id (x), we denote by Bx ,∑wix the total

byte traffic of x. Note that wix is the weight of the i’th packet

of flow x (in bytes).Similarly, Px ,

∑pi=<x,wi>

1 is the packet traffic of x.1) Problem definition: This paper addresses the (ε, δ) -

FREQUENCY ESTIMATION problem. That is, for each flowan algorithm is required to provide an estimator Bx thatapproximates the real byte count Bx in the following manner:

Pr[∣∣∣Bx − Bx∣∣∣ ≤ εB] ≥ 1− δ.

Where B is the total byte traffic from all flows.

B. Architecture and considered scenario

In our considered architecture a network device samplessome of the traffic and transfers it to the measurementalgorithm. The algorithm can either be implemented in avirtual machine or as part of the network device. There aretwo degrees of freedom, the sampling method used and theunderlying measurement algorithm. For example, in our BUS-SS algorithm, BUS is the chosen sampling method and SpaceSaving (SS) is the chosen measurement algorithm. Differentfunctionalities such as load balancing, traffic engineering, andanomaly detection can then use the measurement data duringtheir operation. This scenario is illustrated in Figure 2.

Fig. 2: A network device forwards a sample to a monitoringservice that is used by higher level network functionalities.

C. The Tap: PUS and BUS sample methods

The tap controls what part of the traffic is being measured.In principle, we can forward all traffic to the measurementalgorithm but this is often prohibitively expensive. We considertwo methods: Packet Uniform Sampling (PUS) that is com-monly used in networking devices and Byte Uniform Sampling(BUS) that is suggested in this work. To accurately presentPUS and BUS we require the following notations: Denote bySix a random {0, 1} variable where Six = 1 if the i’th packetof flow x is sampled and 0 otherwise. Sx =

∑Six, is the

number of sampled packets from flow x.1) Legacy PUS method: For completeness, Definition III.1

formally defines the commonly used PUS method where allpackets are sampled with equal probability. We suggest twobyte estimation methods: BYTE-EST-1 that simply multipliesthe packet count estimation with the average packet sizeand BYTE-EST-2 that sums the size of sampled packets andmultiplies the result with the inverse sampling probability.

Definition III.1 ((PUS) Sampling method). 1. Each packet issampled with probability τ .The number of packets (Px) is estimated with the numberof sampled packets (Sx) normalized by the inverse samplingprobability. That is: Px , Sx · τ−1.There are two ways to estimate the byte traffic of a flow (Bx),the first is to multiply the packet estimator with the averagepacket size.3. BYTE-EST(x)-1 returns Bx = Px ·A.Alternatively, we can sum the weight the packets and multiplyby the inverse sample probability.4. BYTE-EST(x)-2 returns Bx = τ−1

∑Six=1 w

ix.

2) BUS method: BUS samples packets with probabilityproportional to their size. That is, larger packets are sampledwith higher probability than smaller ones. The number ofsampled packets in BUS is proportional to the total byte traffic,and we use the parameter β as the sample probability of asingle byte. BUS is formally defined in Definition III.2.

Definition III.2 ((BUS) Sampling method). 1. Packets aresampled with probability proportional to their size: Thesampling probability, depends on (w) in the following manner:

P (w) , βw

BUS estimates the number of bytes in the flow by multiplyingthe number of sampled packets Sx with β−1. That is:Bx = Sx · β−1.BUS is well defined (i.e,P (w) ≤ 1) when β ≤ 1

M .

D. The BUS-SS algorithm

Samples contribute to the operation speed while streamingalgorithms are used to reduce the amount of required space.BUS can be combined with many measurement algorithmsand the result is a weighted algorithm. In BUS the numberof sampled packets is proportional to the byte traffic volume.Hence, the underlying algorithm simply counts the numberof sampled packets. This enables BUS-SS to asymptoticallyimprove the update time for byte volume estimation fromlogarithmic to constant as packet counting algorithms areasymptotically faster.

Every algorithm that satisfies the frequency estimation pro-blem can be used with BUS. Given a flow identifier (x)the algorithm provides an estimator (Sx) for the number ofsampled packets Sx. The estimated traffic volume of x, (Bx)is estimated in the following manner: Bx , Sx · β−1.

BUS-SS has the following merits. (i) It is simple to im-plement as it uses an unmodified standard algorithm. (ii) Itoperates on constant time and only processes a sample of theinput. (iii) It solves the weighted volume estimation problemwhen the traffic volume is large enough.

IV. ANALYSIS

A. BUS sampling method analysis

We start by showing that BUS is unbiased. Formally, weneed to show that:

E(Bx) = Bx,

where Bx is defined in Definition III.2.

Theorem IV.1. BUS is unbiased. Namely, E(Bx) = Bx.

Proof. Recall that Bx = Sx ·β−1 and therefore it is sufficientto show that E(Sx) = βBx.

E(Sx) = E(∑

Six

)=∑

E(Six)

For each Six, the sampling probability is P (wix) = β · wix.Therefore:∑

E(Six) =∑

β · wix = β ·(∑

wix

)= β ·Bx

Therefore, E(Sx) = βBx and we get that:

E(Bx) = E(Sx · β−1) = E(Sx) · β−1 = Bxββ−1 = Bx

Theorem IV.1 shows that BUS is unbiased. Our next stepis to be able to compare BUS and PUS on the same numberof sampled packets. Toward that end, we need to establishthat the average packet sample probability (τ ) has the samemeaning for both methods. Formally for BUS we need to finda τ s.t: E(S) = τ · N , where S is the number of sampledpackets from all flows.

Theorem IV.2. Let τ , β ·A, then E(S) = τN

Proof. Theorem IV.1 shows that:

E(B) = E(S · β−1

)= B

Since B = N ·A, we get:

E (S) = βB =τ

AB =

τ

ANA = τN.

Theorem IV.2 allows us to compare BUS to PUS on equalsize samples, if we also know the average packet size (A).Note that the knowledge of A is only required to evaluate PUSand BUS on equal terms. Specifically, BUS does not requireany prior knowledge of A.

Intuitively, there is a complex relation between the requiredaccuracy (ε), the required confidence (δ), the sample proba-bility (β), and the total byte volume (B). Therefore, if weset ε, δ and β we can derive an upper bound on the minimalstream volume that is required to solve (ε, δ) - FREQUENCYESTIMATION. For simplicity, we use result 4.9 in [37]: In thisresult, our goal is to estimate the unknown average successprobability (p) of a random variable (Sx) that is the sum ofmultiple independent Poisson trails. The trails (Six) only needto be independent and do not need to have the same successprobability. In our case, BUS sample packets independentlyof each other and therefore Six are independent of each other.The result states that:

Pr (p /∈ [p− ε, p+ ε]) ≤ e−nε2

2p + e−nε2

3p . (1)

We now explain how BUS matches the terms of Equation 1:

Definition IV.1 (BUS adaptation of Eq. 1).• p , E(Sx)

Bx= β, the probability that a byte belongs to a

sampled packet is β.• p = Sx

Bxis the measured byte sample probability.

• n = Bx.

Given the sampling error εs, sampling confidence δs andbyte sample probability β, our main theorem bounds requiredtraffic for BUS to provide its formal guarantee

Theorem IV.3. BUS solves (εs, δs) - FREQUENCY ESTIMA-TION for B > 6

5 ln(

1δs

)ε−2s β−1.

Proof. We first need to bring the Equation 1 to match ourformal problem. We apply the adaptation in Definition IV.1 ofEquation 1 and get:

Pr

(SxBx

/∈[E (Sx)

Bx− εsβ,

E (Sx)

Bx+ εsβ

])= Pr (Sx /∈ [E (Sx)− εsβBx,E (Sx) + εsβBx])

Note that: Sx · β−1 = Bx and E(Sx) · β−1 = Bx hence:

= Pr(∣∣∣Bx − Bx∣∣∣ ≥ εsBx) ≤ Pr

(∣∣∣Bx − Bx∣∣∣ ≥ εsB)Now that the left-hand side of Equation 1 is phrased correctlywe can apply Definition IV.1 to the right-hand side and get:

Pr(∣∣∣Bx − Bx∣∣∣ ≥ εsB) < e

(−ε2sβ

2Bx2β

)+ e

(−ε2sβ

2Bx3β

). (2)

Equation 2 requires us to know Bx, (or an upper bound onBx). We use the trivial observation that Bx < B to continue.

Pr(∣∣∣Bx − Bx∣∣∣ ≥ εsB) ≤ e

(−ε2sβB

2

)+ e

(−ε2sβB

3

).

We denote δs = e

(−ε2sβB

2

)+ e

(−τNε2sβB

3

)and solve for B.

ln

(1

δs

)=ε2sβB

2+ε2sβB

3.

B =6

5ln

(1

δs

)ε−2s β−1. (3)

Therefore, for byte traffic volumes larger than B wehave: Pr

(∣∣∣Bx − Bx∣∣∣ ≥ εB) ≤ δs, and therefore:

Pr(∣∣∣Bx − Bx∣∣∣ ≤ εB) ≥ 1− δs, completing the proof.

In some cases, the measurement interval is known in ad-vance, and for such measurements it may be useful to derivean upper bound on the minimal β that enables BUS to solvethe (ε, δ) - FREQUENCY ESTIMATION problem for the givenmeasurement size.

Corollary IV.4. Given εs, δs and B, BUS solves (εs, δs) -

FREQUENCY ESTIMATION for β >6 ln( 1

δs)ε−2s

5B

Proof. The proof is derived by extracting β as function of B,in Equation 3.

B. BUS-SS analysis

We now analyze BUS-SS, where the Space Saving algo-rithm (SS) receives a BUS sample instead of the originalstream. In BUS-SS, part of the error comes from BUS (εs)and the rest from SS (εa). We name the sampling error εs andthe approximation error εa.

The error of streaming algorithms depends on the numberof sampled packets and hence, we first need to bound the sizeof the entire BUS sample.

Corollary IV.5. Consider a BUS sample with probability β,and stream of volume B ≥ 6

5 ln(

1δs

)ε−2s β−1, then

Pr (S ≤ βB (1 + εs)) ≥ 1− δs.

Proof. Note that the expected number of sampled packets isE(S) = β · B, and that B ≥ 6

5 ln(

1δs

)ε−2s β−1. Therefore,

we use Theorem IV.3 for the entire sample to complete theproof.

Using Corollary IV.5 we can guarantee that although thenumber of sampled packets fluctuates, the actual sample sizeis bounded. Given an error parameter ε′a and an algorithm Athat solves the (εa, δa) - FREQUENCY ESTIMATION problem,we define:

εa ,ε′a

1 + εs.

According to Corollary IV.5, with probability 1−δ the numberof sampled packets is at most (1+ εs)βB and the accuracy isas required.

Theorem IV.6. Consider BUS-SS where an algorithm (A) thatsolves the (ε, 0) - FREQUENCY ESTIMATION problem receivesa BUS sample as input. If B > 6

5 ln(

1δs

)ε−2s β−1, then

Pr[∣∣∣Bx − Bx∣∣∣ ≤ εB] ≥ 1− δ, for ε = εa + εs and δ = 2δs.

That is, BUS-SS solves (ε, δ) - FREQUENCY ESTIMATION.

Proof. Algorithm A solves the εa frequency estimation pro-blem. That is, it provides an estimator Sx for Sx, the numbersampled packets from flow x. A is deterministic and hence:

Pr(∣∣∣Sx − Sx∣∣∣ ≥ εaβB) = 0. (4)

B > 65 ln

(1δs

)ε−2s β−1 and according to Theorem IV.3:

Pr(∣∣∣Bx − Bx∣∣∣ ≥ εsB) ≤ δs (5)

According to definition III.2: Bx = Sx ·β−1. We need to showthat:

Pr(∣∣∣Bx − Sxβ−1

∣∣∣ ≥ εB) ≤ δsIn order to do so, we rewrite the above as:

= Pr(∣∣∣Bx − Bx + Bx − Sxβ−1

∣∣∣ ≥ εsB)≤ Pr

(∣∣∣Bx − Bx∣∣∣+ ∣∣∣Bx − Sxβ−1∣∣∣ ≥ εB)

Since Bx = Sxβ−1 we can write the above as:

= Pr(∣∣∣Bx − Bx∣∣∣+ ∣∣∣Sxβ−1 − Sxβ−1

∣∣∣ ≥ εB) (6)

Note that: Pr(∣∣∣Sxβ−1 − Sxβ−1

∣∣∣ ≥ εaB) = 0

Due to Equation 4. Pr(β−1

∣∣∣Sx − Sx∣∣∣ ≥ εaB) = 0 Wecan reason about Equation 6 in the following manner:

(6) ≤ Pr(∣∣∣Bx − Bx

∣∣∣ ≥ εsB)+ Pr

(∣∣∣Sxβ−1 − Sxβ−1∣∣∣ ≥ εaB

)= Pr

(∣∣∣Bx − Bx∣∣∣ ≥ εsB)+ 0 ≤ δs (7)

That is, we showed that: Pr(∣∣∣Bx − Sxβ−1

∣∣∣ ≥ εB) ≤ δs

and therefore: Pr(∣∣∣Bx − Sxβ−1

∣∣∣ ≤ εB) ≥ 1 − δs, ac-counting for over sample and applying union bound yieldsPr(∣∣∣Bx − Sxβ−1

∣∣∣ ≤ εB) ≥ 1− δ completing the proof.

Theorem IV.6 shows the correctness of BUS-SS.

V. EXPERIMENTAL EVALUATION

A. Metrics and datasets

Our evaluation includes four datasets of a mix of UDP/TCPand ICMP packets collected from major backbone routers inboth Chicago [38], [39] and San Jose [14], [15] in the years2014-2016. We also include a trace of TCP packets from theUCLA Computer Science department [40] in the year 2001.In these datasets, the percentage and volume of large TCPpackets varies, in the San Jose traces [14], [15] less than 1%of the packets account for up to 25% of the traffic volume. Inthe Chicago datasets large TCP packets are insignificant, yetin both datasets these packets become increasingly common.In the UCLA trace, there are no large tcp packets probablybecause the technological state of 2001.

On-Arrival volume estimationWe use the On-Arrival error model that is also used by [17],[41]. That is, a query is performed before every packet arrival.We then measure the Root Mean Square Error (RMSE) of thealgorithm, i.e.,

RMSE(Alg)∆=

√√√√ 1

|S|

S∑t=1

(Bxt −Bxt)2,

where xt is the identifier of the t′th packet.

B. BUS sampling method evaluation

We start with the evaluation of BUS and compare it tothe commonly available PUS. For a fair comparison, wemeasured the average packet size (A) in each trace, and usedTheorem IV.2 to assure that both BUS and PUS sample onaverage the same number of packets. We compare BUS toboth methods suggested in Definition III.1. PUS1 utilizesBYTE-EST-1, and multiplies the packet estimation with A.While PUS2, utilizes the method BYTE-EST-2 that maintains

(a) SanJose14 (b) UCLA01 (c) Chicago16

(d) SanJose13 (e) Legend (f) Chicago15

Fig. 3: The effect of the packet sampling probability (τ ) on estimation error.

a weighted sample. Each experiment was ran 10 times andaverages are reported.

Figure 3 shows that PUS1 is significantly worse than thealternatives. Motivating the need for byte sampling techniques.It has error rates that vary between ≈ 20 MB RMSE inChicago 16 (Figure 3c) and over 1GB RMSE in San Jose14 (Figure 3a). PUS2 offers a considerable improvement overPUS1 but is still inferior to BUS in all tested workloads.

The difference between PUS2 and BUS is workload depen-dent. In Chicago15 and Chicago16 (Figure 3f and 3c), thedifference is relatively small (≈ 10%). However,in SanJose13and SanJose14 (Figure 3d and 3a), the difference is verylarge and BUS is up to x100 more accurate than PUS2.The explanation for this lies in the characteristics of traffic.Specifically, the frequency and combined volume of large TCPpackets in each trace. In contrast, BUS offers a predictableaccuracy in all of the traces.

C. BUS-SS Evaluation

We now evaluate BUS-SS and compare it to ’Vanilla SS’;that is, an unmodified SS implementation that operates inlogarithmic complexity and processes the entire stream. Wealso compare to PUS2-SS, where a PUS2 weighted sampleis fed to a Vanilla SS. PUS2-SS operates in logarithmiccomplexity as well. In comparison, BUS-SS relies on animplementation of SS that cannot perform weighted updates,but operates in constant time. In addition, all algorithms areallocated the same amount of space.

The results in Figure 4 show that for all datasets exceptUCLA01, BUS-SS behaves very similar to Vanilla SS. Whenthe packet sample probability is too low, the error of BUS-SSis dominated by the sample error and is considerably higherthan that of Vanilla SS. It is encouraging to observe that the’break-away’ point still allows for sampling rates of about2−9 in the San Jose traces and 2−8 in the Chicago and UCLAtraces.

VI. OPEN VSWITCH IMPLEMENTATION

This section describes our implementation of BUS and PUSin DPDK enabled Open vSwitch (OVS). We start by providinga high level overview for the OVS architecture and continue bydescribing our implementation and the performed evaluation.

A. Background and overview

Virtual switching is a key building block of any virtualizedenvironment which enables flexible interconnection of virtualhosts. Such flexibility is fundamental to NFV as it allowschaining virtual network functionalities together. Operationspeed is the key bottleneck of virtual switching as it competeswith dedicated and custom made hardware.

Context switches between user and kernel spaces cause per-formance bottlenecks. Specifically, the routing of each arrivingpacket requires a context switch to reach the physical networkinterface card (NIC) and its data needs to be physically copiedfrom the NIC’s buffer to a user space buffer. In practice,high-speed virtual switching is enabled by many software and

(a) SanJose14 (b) UCLA01 (c) Chicago16

(d) SanJose13 (e) Legend (f) Chicago15

Fig. 4: Effect of the packet sample probability (τ ) on empirical error. Space requirement is equal for all algorithms (εa = 0.001).

hardware acceleration technologies such as NetMap [42] andIntel’s Data Plane Developer Kit (DPDK) [43] that reduce thenumber of context switches and the amount of copied data.

DPDK allows OVS to perform the entire packet processingpipeline in user space and thereby avoiding the aforementionedoverheads. It consists of a set of data plane libraries thatoptimize packet processing. DPDK takes advantage of PollMode Drivers (PMD) for user space packet processing anddirect access to NIC buffers.

Therefore, our implementation targets the DPDK versionof OVS. That version is designed as a multilayer softwareswitching implementation composed of two main components:ovs-vswitchd1 and ovsdb-server. Due to space constraints, wefocus on vswitchd component and refer the interested readerto [44] for additional information.

The vswitchd module implements both the control and thedata plane in user space. Network packets ingress the datapath(dpif or dpif-netdev) either from a physical port connectedto the physical NIC or from a virtual port connected to aremote host (e.g., a virtual network function). Then, for eachpacket, the datapath parses the headers and determines the setof actions to be applied (e.g., forwarding or rewriting a specificheader).

1Note that the datapath is not integrated with vswitchd in the Kernel OVS.

B. BUS OVS implementation

Currently the OVS project does not have datapath samplingcapabilities. The existing sampling mechanisms, such as sFlowand NetFlow, are not designed to communicate with physicaland virtual interfaces. Thus, sampled traffic is sent to a userspace application that is unable to send traffic back to OVS2.Therefore, we had to implement such capabilities for bothphysical and virtual ports. Our implementation is realizedthrough a simple sampled-based forwarding action.

Our BUS implementation extends the DPDK accelerateddatapath capabilities and is limited to the dpif-netdev compo-nent. As packets are processed through the datapath, we retaineasy access to their metadata and specifically to the payloadsize field that is required by BUS. Our BUS implementationredefines a forwarding action as a ’branch’ of the datapath.Sampled packets are actually forwarded and unsampled pac-kets are dropped. Hence, our core implementation is simpleand only requires few lines of code in dpif-netdev.

C. OVS Evaluation

1) Environment setup: We evaluate BUS on an NFV-basedenvironment. Our evaluation consists of two HP ProLiantservers. Each is equipped with an Intel Xeon E3-1220v2

2OVS DPDK owns the management of all interface attached to it. Additio-nally, current implementation does not allow to attach kernel-owned interfacesto OVS DPDK – which would allow user space application to communicatewith DPDK interfaces

Fig. 5: Our evaluated scenario: Traffic passes through twoVNFs and a portion of it is also sent to the monitoring service.

processor, with 4 physical cores at 3.1 Ghz. The servers have8 GB RAM, and a DPDK-enabled Intel 82599ES 10 Gbit/snetwork device with two interfaces (physical ports) that aredirectly interconnected. The servers are installed with CentOSversion 7.2.1511, Linux kernel version 3.10.0.

One server is used as Design Under Test (DUT), and theother is used as a traffic generator. We use Open vSwitch 2.6compiled with Intel’s DPDK 16.07. All VNFs are configuredwith Fedora 22 operating system, Linux kernel version 4.0.4– running on top of qemu version 2.6 and are configured tohave a single virtual CPU pinned to a specific physical coreand 512MB of RAM. Packet generation is performed on thetraffic generator server that sends traffic through one networkinterface and receives processed traffic from the DUT server onthe other network interface. We generate UDP network trafficwith packet sizes that are determined by the real packet traces,the traffic is generated with MoonGen traffic generator [45].

2) Scenario: Our use case is an NFV-based service chai-ning deployment with a monitoring VNF. The service chainhas two VNFs in line which are forwarding the receivednetwork traffic back to the traffic generator. Additionally, thetraffic sent out by the second VNF is sampled to a thirdmonitoring service. Note that this evaluation measures OVSthroughput and not the performance of the monitoring service.Figure 5 depicts the aforementioned scenario.

3) Results: Figure 6 shows the average end-to-end throug-hput for the SanJose14 trace, for PUS and BUS compared tothe two baselines. The baseline on the left shows throughoutwhen all the packets are sampled and the one to the rightshows the vanilla performance when no sampling takes place.As can be observed, BUS and PUS achieve similar throughputfor all sampled probabilities; further we achieve 20% betterthroughput in comparison to the baseline where all the trafficis sampled. Additionally, our implementation still has nonnegligible overheads as performance is improved by 9% whennot sampling at all. This indicates that there is still room foroptimizations. Similar results were obtained on other traces.

Fig. 6: Average measured throughput in our NFV use case.

Algorithm Space Query Update

Space Saving [26] O(1ε

)O(1) O

(log 1

ε

)Frequent [46] O

(1ε

)O(1) O

(log 1

ε

)CM Sketch [18] O

(1ε· log 1

δ

)O

(log 1

δ

)O

(log 1

δ

)Count Sketch [19] O

(1ε2

· log 1δ

)O

(log 1

δ

)O

(log 1

δ

)BUS - SS O

(1ε

)O(1) O(1)

TABLE II: Positioning of BUS-SS among other popular flowvolume estimation algorithms. Space Saving and Frequent aredeterministic, while CM Sketch, Count Sketch and BUS-SSare probabilistic.

VII. CONCLUSIONS

This work is about estimating per-flow byte volume withsamples. Such a capability is needed by network functiona-lities but the applied uniform sampling method is sometimesinefficient due to large variation in packets’ size. We showedthat this is indeed a problem in recent Internet traces.

We introduced an alternative and sampling method calledBUS that samples packets with probability proportional totheir byte size. We also introduced a measurement archi-tecture where a network device transmits a BUS sample toa virtual machine that perform per-flow measurement. Weshowed that BUS enables the underlying algorithm simplycount the number of sampled packets and still provide reli-able estimations for per-flow byte volume. This enables theuse of asymptotically faster algorithms. Specifically, whenthe underlying measurement algorithm is Space Saving thecombined algorithm (BUS-SS) achieves constant update time.This improves existing works that require logarithmic updatetime. Table II compares BUS-SS to previous works.

We rigorously analyzed BUS and BUS-SS and provedstrong accuracy guarantees. Our algorithms require a minimalamount of traffic to be included in the measurement. This isa minor limitation as modern link technologies dramaticallyincrease the throughput. Such enhancements can be leveragedto enable shorter measurements or more aggressive sampling.

We evaluated BUS and BUS-SS using five real Internet

traces. We showed that for the same sample size BUS is alwaysmore accurate than uniform sampling. Specifically, the latterrequires up to x50 times as many packets (and bytes) to matchthe accuracy of BUS as shown in Figure 3a. We then showedthat the accuracy of BUS-SS is similar to that of its underlyingalgorithm even when sampling less than 1% of the traffic.

Finally, we implemented BUS in the DPDK-enabled OpenvSwitch (OVS), a real open source virtual switch. Our im-plementation allows sampled traffic to be transmitted directlyto virtual machines and thus eases the use of virtual networkfunctionalities. We evaluated a distributed implementation ofBUS-SS where Space Saving is placed in a virtual machineand OVS samples packets according to BUS and forwardsthem to the measurement machine. We showed that BUS isindeed practical as OVS achieves similar throughput with BUSand with the commonly used uniform packet samples. BUS isreleased as branch of the Open vSwitch project, and is opensourced as well [47].

ACKNOWLEDGEMENT

The authors would like to thank Ran Ben Basat for hisinsightful comments in early stages of this work.

REFERENCES

[1] A. Kabbani, M. Alizadeh, M. Yasuda, R. Pan, and B. Prabhakar, “Af-qcn: Approximate fairness with quantized congestion notification formulti-tenanted data centers,” in IEEE HOTI, 2010, pp. 58–65.

[2] T. Benson, A. Anand, A. Akella, and M. Zhang, “Microte: Fine grainedtraffic engineering for data centers,” in ACM CoNEXT, 2011, p. 8.

[3] P. Garcia-Teodoro, J. E. Diaz-Verdejo, G. Macia-Fernandez, and E. Vaz-quez, “Anomaly-based network intrusion detection: Techniques, systemsand challenges,” Computers and Security, pp. 18–28, 2009.

[4] L. Ying, R. Srikant, and X. Kang, “The power of slightly more thanone sample in randomized load balancing,” in IEEE INFOCOM 2015,April, pp. 1131–1139.

[5] M. Alizadeh, T. Edsall, S. Dharmapurikar, R. Vaidyanathan, K. Chu,A. Fingerhut, V. T. Lam, F. Matus, R. Pan, N. Yadav, and G. Varghese,“Conga: Distributed congestion-aware load balancing for datacenters,”ser. ACM SIGCOMM 2014, pp. 503–514. [Online]. Available:http://doi.acm.org/10.1145/2619239.2626316

[6] G. Einziger and R. Friedman, “Tinylfu: A highly efficient cache admis-sion policy,” in Euromicro PDP, 2014, pp. 146–153.

[7] N. Hua, B. Lin, J. J. Xu, and H. C. Zhao, “Brick: A novel exact activestatistics counter architecture,” ser. ACM/IEEE ANCS, 2008, pp. 89–98.

[8] D. Shah, S. Iyer, B. Prabhakar, and N. McKeown, “Maintaining statisticscounters in router line cards.” IEEE Micro, pp. 76–81, 2002.

[9] S. Ramabhadran and G. Varghese, “Efficient implementation of astatistics counter architecture,” ACM SIGMETRICS, pp. 261–271, 2003.

[10] Y. Lu, A. Montanari, B. Prabhakar, S. Dharmapurikar, and A. Kabbani,“Counter braids: a novel counter architecture for per-flow measurement,”in ACM SIGMETRICS, 2008, pp. 1799–1807.

[11] M. Chen and S. Chen, “Counter tree: A scalable counter architecturefor per-flow traffic measurement,” in IEEE ICNP, 2015, pp. 111–122.

[12] “Sflow’s homepage, http://sflow.org.”[13] “Random sampled netflow. http://www.cisco.com.”[14] P. Hick, “CAIDA Anonymized 2014 Internet Trace, equinix-sanjose

2013-06-19 13:00-13:05 UTC, Direction B.”[15] ——, “CAIDA Anonymized 2013 Internet Trace, equinix-sanjose 2013-

12-19 13:00-13:05 UTC, Direction B.”[16] E. Tsidon, I. Hanniel, and I. Keslassy, “Estimators also need shared

values to grow together,” in IEEE INFOCOM, 2012, pp. 1889–1897.[17] R. Ben-Basat, G. Einziger, R. Friedman, and Y. Kassner, “Heavy hitters

in streams and sliding windows,” in IEEE INFOCOM, 2016.[18] G. Cormode and S. Muthukrishnan, “An improved data stream summary:

The count-min sketch and its applications,” J. Algorithms, pp. 58–75,2005.

[19] M. Charikar, K. Chen, and M. Farach-Colton, “Finding frequent itemsin data streams,” in EATCS ICALP, 2002, pp. 693–703.

[20] A. Metwally, D. Agrawal, and A. E. Abbadi, “Efficient computation offrequent and top-k elements in data streams,” in IN ICDT, 2005.

[21] R. M. Karp, S. Shenker, and C. H. Papadimitriou, “A simple algorithmfor finding frequent elements in streams and bags,” ACM TransactionsDatabase Systems, vol. 28, no. 1, Mar. 2003.

[22] T. Li, S. Chen, and Y. Ling, “Per-flow traffic measurement throughrandomized counter sharing,” IEEE/ACM Trans. on Networking, 2012.

[23] G. S. Manku and R. Motwani, “Approximate frequency counts over datastreams,” in VLDB, 2002.

[24] X. Dimitropoulos, P. Hurley, and A. Kind, “Probabilistic lossy counting:An efficient algorithm for finding heavy hitters,” ACM SIGCOMM,vol. 38, no. 1, Jan. 2008.

[25] Q. Rong, G. Zhang, G. Xie, and K. Salamatian, “Mnemonic lossy coun-ting: An efficient and accurate heavy-hitters identification algorithm,” inIEEE IPCCC, 2010, pp. 255–262.

[26] G. Cormode and M. Hadjieleftheriou, “Finding frequent items in datastreams,” VLDB, vol. 1, no. 2, pp. 1530–1541, Aug. 2008.

[27] ——, “Methods for finding frequent items in data streams,” J. VLDB,vol. 19, no. 1, pp. 3–20, 2010.

[28] N. Manerikar and T. Palpanas, “Frequent items in streaming data: Anexperimental evaluation of the state-of-the-art,” Data Knowl. Eng., pp.415–430, 2009.

[29] B.-Y. Choi, J. Park, and Z.-L. Zhang, “Adaptive random sampling forload change detection,” ACM SIGMETRICS, pp. 272–273, 2002.

[30] C. Estan, K. Keys, D. Moore, and G. Varghese, “Building a betternetflow,” ACM SIGCOMM, 2004.

[31] N. Duffield, C. Lund, and M. Thorup, “Flow samplingunder hard resource constraints,” ser. ACM SIGMETRICS2004/Performance 2004, pp. 85–96. [Online]. Available:http://doi.acm.org/10.1145/1005686.1005699

[32] ——, “Charging from sampled network usage,” in Proceedings of the 1stACM SIGCOMM Workshop on Internet Measurement, 2001, pp. 245–256. [Online]. Available: http://doi.acm.org/10.1145/505202.505232

[33] A. Andoni, R. Krauthgamer, and K. Onak, “Streaming algorithms viaprecision sampling,” in IEEE FOCS, 2011, pp. 363–372.

[34] N. Duffield, “Fair sampling across network flow measurements,” ACMSIGMETRICS Perform. Eval. Rev., pp. 367–378, Jun. 2012. [Online].Available: http://doi.acm.org/10.1145/2318857.2254800

[35] G. Einziger, B. Fellman, and Y. Kassner, “Independent counter estima-tion buckets,” in IEEE INFOCOM, 2015, pp. 2560–2568.

[36] L. Yang, W. Hao, P. Tian, D. Huichen, L. Jianyuan, and L. Bin,“Case: Cache-assisted stretchable estimator for high speed per-flowmeasurement,” in IEEE INFOCOM, 2016.

[37] M. Mitzenmacher and E. Upfal, Probability and Computing: Rando-mized Algorithms and Probabilistic Analysis. New York, NY, USA:Cambridge University Press, 2005.

[38] P. Hick, “CAIDA Anonymized 2015 Internet Trace, equinix-chicago2015-12-17 13:00-13:05 UTC, Direction A.”

[39] ——, “CAIDA Anonymized 2016 Internet Trace, equinix-chicago 2016-02-18 13:00-13:05 UTC, Direction A.”

[40] “Unpublished, see http://www.lasr.cs.ucla.edu/ddos/traces/.”[41] R. Ben-Basat, G. Einziger, R. Friedman, and Y. Kassner, “Randomized

admission policy for efficient top-k and frequency estimation,” CoRR,vol. abs/1612.02962, 2016.

[42] L. Rizzo, “Netmap: A novel framework for fast packet i/o,” in Procee-dings of the 2012 USENIX Conference on Annual Technical Conference,ser. USENIX ATC’12. Berkeley, CA, USA: USENIX Association,2012, pp. 9–21.

[43] “Intel dpdk,”http://dpdk.org/, accessed: 04-19-2016.

[44] B. Pfaff, J. Pettit, T. Koponen, E. Jackson, A. Zhou, J. Rajahalme,J. Gross, A. Wang, J. Stringer, P. Shelar, K. Amidon, and M. Casado,“The design and implementation of open vswitch,” in USENIX NSDI2015.

[45] P. Emmerich, S. Gallenmuller, D. Raumer, F. Wohlfart, and G. Carle,“Moongen: A scriptable high-speed packet generator,” in ACM IMC2015, pp. 275–287.

[46] R. Berinde, P. Indyk, G. Cormode, and M. J. Strauss, “Space-optimalheavy hitters with strong error bounds,” ACM Trans. Database Syst.,vol. 35, no. 4, p. 26, 2010.

[47] “Byte Uniform Sampling - Open vSwitch Implementation, available:https://github.com/ovssample/bus .”