IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 8, NO. 5, OCTOBER 2006 1005

Efficient Streaming Packet Video OverDifferentiated Services Networks

Fei Zhang, James Macnicol, Member, IEEE, Mark R. Pickering, Member, IEEE, Michael R. Frater, Member, IEEE,and John F. Arnold, Senior Member, IEEE

Abstract—We investigate streaming video over DifferentiatedServices (Diffserv) networks that can provide a number of ag-gregated traffic classes with increasing quality guarantee. Wepropose a method to measure the loss impact of a video packet onthe quality of the decoded video images. We show how the optimalQuality-of-Service (QoS) mapping from the video packets into aset of traffic classes depends on the loss rates of the classes andthe pricing model, and we develop an algorithm to accurately findthe optimal QoS mapping. The performance of our algorithm isevaluated through computer simulations and compares favorablyto an existing algorithm.

Index Terms—Differentiated services networks, error resilience,video compression.

I. INTRODUCTION

I N RECENT years, there has been a growing demand forstreaming multimedia applications over the Internet. In

particular, video communication over the Internet has at-tracted much research interest. Providing good image qualityin real-time or near real-time video communications such asvideo-conferencing, video-telephony, and video-on-demand,is difficult with the current “best effort” Internet model. Thissame-service-to-all paradigm has become increasingly inade-quate for Internet applications (including streaming video)thathave diverse Quality-of-Service (QoS) requirements. In thiseffort, the Internet Engineering Task Force (IETF) has pro-posed two distinct approaches for service differentiation: theIntegrated Services (Intserv) [1] and Differentiated Services(Diffserv) [2] architectures. However the Intserv architecturehas received very limited acceptance among the networkcommunity due to its problem of nonscalability and nonman-ageability [3]. On the other hand, the Diffserv approach is morerecent, and its main goal is to provide a scalable and manage-able network with service differentiation capability. Unlike theIntserv approach where each router tracks individual flows, thecore of a Diffserv network only distinguishes between packetsmarked as belonging to different aggregate flows, also known

Manuscript received March 24, 2003; revised November 21, 2005. Thiswork was supported in part by the Australian Research Council under GrantA00104347. The associate editor coordinating the review of this manuscriptand approving it for publication was Dr. Chang Wen Chen.

TF. Zhang, M. R. Pickering, M. R. Frater, and J. F. Arnold are with theSchool of Information Technology and Electrical Engineering, AustralianDefence Force Academy, The University of New South Wales, Canberra, ACT2600, Australia (e-mail: f.zhang@adfa.edu.au; m.pickering@adfa.edu.au;m.frater@adfa.edu.au, j.arnold@adfa.edu.au).

J. Macnicol, deceased, was with the School of Information Technology andElectrical Engineering, Australian Defence Force Academy, The University ofNew South Wales, Canberra, ACT 2600, Australia.

Digital Object Identifier 10.1109/TMM.2006.879865

as service classes. No absolute performance guarantees areprovided in this type of network, only that aggregates withhigher priority will receive preferential treatment to those oflower priority. In this work we only consider Diffserv AssuredForwarding [4] networks.

One of the key questions that must be addressed in thedevelopment of Diffserv network architectures is how to effi-ciently utilize a given Diffserv network model for multimediastreaming. In this effort, a number of research groups haveinvestigated video streaming over simulated Diffserv networks[5]–[13]. Our new contributions are as follows.

• We propose a simple method to measure the loss impactof a video packet on the decoded video quality when thevideo sequence under consideration has periodic I-frames,taking into account error propagation effects in inter-codedframes.

• We develop a new algorithm to find the solution to the QoSmapping problem, i.e., how to assign video packets to theavailable Diffserv classes.

• We demonstrate through simulation that the new QoS map-ping algorithm is more effective in minimizing the totalloss impact of a video packet stream that passes through aDiffserv network, and therefore, can produce better videoquality than the existing QoS mapping algorithm of Shinet al. [6].

The rest of this paper is organized as follows. In Section II,we discuss how to measure the loss impact of a video packet.In Section III, we derive the optimal QoS mapping algorithm.Simulation results are presented in Section IV and some con-cluding remarks are made in Section V.

II. PACKET LOSS IMPACT ON VIDEO QUALITY

In order to use a Diffserv network effectively, it is necessaryto know the loss impact of every video packet, so that thosepackets with higher loss impact may be sent via an appropriatehigher priority traffic class that suffers lower loss probability.(In this paper, we only consider the loss effect. We assume thatdelay and jitter effects can be absorbed by using a large playbuffer in the decoder.) There are many methods available for es-timating packet loss impact with various levels of computationalcomplexity that could be substituted for the technique describedbelow, see [14] for a recent overview.

To measure the loss impact of a video packet quantitatively,we assume that the decoder is using a normative error conceal-ment algorithm. For the first I-frame temporal concealment isnot possible and spatial interpolation must be used. For sub-sequent frames we employ a temporal concealment scheme byusing a neighboring MB’s motion vectors, if available, or zero

1520-9210/$20.00 © 2006 IEEE

1006 IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 8, NO. 5, OCTOBER 2006

Fig. 1. Optimal QoS mapping algorithm based on recursive relation (9).

motion vector if no reliable neighboring macroblock is avail-able. Under this error concealment scheme, if a packet is lost inthe th frame the initial error can be calculated as

(1)

where the sum is over all the corrupted pixels, and is thedifference between the decoded signals with and without theerror due to the packet loss. In other words, is the differencebetween the correctly decoded texture for frame and the outputof the error concealment algorithm attempting to recover fromthe loss of the packet.

This initial error will propagate into the next frame whencorrupted pixels are used for motion-compensated prediction:

(2)

where is the number of times the corrupted pixel is refer-enced by the th frame. This temporal error propagationwill only be stopped by macroblock intra-refresh or an I-frame.

We consider a compressed video sequence with periodicI-frames. If we assume that , then the total squarederror distortion due to the packet loss in a frame, which isframes away from the immediately previous I-frame, would be

, where is the distance between adjacentI-frames. Therefore, we define the loss impact of a given packetin the th frame as

(3)

This loss impact measurement shows that a packet loss in anI-frame will generally result in a greater (negative) impact onthe decoded video quality, because the initial error has a longpropagation effect. The present method is simpler and requiresless computational effort than that proposed in [6], where threefactors were taken into account in determining the relative pri-ority index of a video packet. The three factors are the initialerror , the average motion vector size, and the number ofintra macroblocks contained in a packet, with weights of 0.7,0.15, and 0.15 respectively. In Section IV, we will compare theperformance of these two methods.

III. OPTIMAL QOS MAPPING ALGORITHM

Once each video packet has been assigned a loss impact (de-noted as ), we consider how to send the packet stream througha relatively differentiated services network that consists of a setof Diffserv classes numbered as . Classexperiences a packet loss rate and attracts a per-packet price

. If the classes are ordered by increasing pricetherefore the losses in each class must be related by

. We assume the sender has accurate knowl-edge of the network state e.g. by receiving feedback from thereceiver.

Under a given price constraint, there are many possiblemappings from the packets to the Diffserv classes. From thesemappings we want to find an optimal one that minimizes thetotal loss impact, and consequently maximize the receivedvideo quality. Here the total loss impact can be expressedmathematically as

(4)

where denotes the Diffserv class to which the th packetis mapped. The price constraint is expressed as

where is the per packet price budget.We note that this constrained optimization problem was also

formulated in [6]–[8], where the optimal solution was calledthe optimal QoS mapping. However, they used a coarse grainedmethod to solve the optimization problem. The packets werefirst classified into a number of categories with each assigned anaverage relative priority index (loss impact). The optimal map-ping from the categories to Diffserv classes was then searchedfor. In the following we will show how to solve the optimizationproblem directly using a Lagrange multiplier approach.

We first sort the packets into ascending order according tothe loss impact. Then it can be shown that the optimal mappingmust be an ordered mapping such that the first packets aremapped to the lowest priority Diffserv class, the nextpackets to the second lowest level Diffserv class, and so on. Insuch a scenario, the total loss impact can be expressed as

(5)

And the price constraint becomes:

(6)

Therefore, we are faced with an optimization problem withinteger decision variables

. The feasible solution is defined by the price con-straint (6). To deal with the constraint we combine (5) and (6)to form the Lagrangian:

(7)

ZHANG et al.: EFFICIENT STREAMING PACKET VIDEO 1007

At the maximum value of the constrained optimizationproblem all partial derivatives of the Lagrangian with respectto the variables are zero:

(8)

Note that is the packet loss impact at the margin betweenclasses and . Eliminating the parameter , we obtain therecursive relation:

(9)

The number of packets in each class must then be set sothe corresponding loss impact satisfies (9). (9) does notcompute , this must be set so that the total cost constraint(6) is met. Fig. 1 outlines a simple search procedure to find themaximum value of that satisfies the constraint, hence maxi-mizing the total sequence quality. This algorithm computes theloss impact at the margin, the corresponding packet indexcan be easily computed by inverting the loss function .

Following from (9) and the fact that the packet loss impactsare sorted in ascending order (so , there is thefollowing additional constraint on prices given the loss ratesexperienced

(10)

IV. SIMULATION RESULTS

To evaluate the performance of the new packet loss impactestimation and QoS mapping algorithms a number of experi-ments were conducted using a standard MPEG-4 encoder and adecoder that had been modified to implement appropriate errorconcealment. Two standard sequences (CIF, 10 fps, 10 s du-ration) were tested: “Foreman” at 320 kbps and “Mother &Daughter” at 160 kbps with a video packet size of 500 bytes.

These packetized bitstreams were then mapped to four Diff-serv classes using the algorithm described in Section III and thetechnique described in [6]. As in [6], we use linearly increasingprices ( without loss of generality)and loss rates of the form . The numberof available Diffserv classes is network-specific and may not befour as used here, these issues are outside the scope of this paper.

The first feature of note is that the new iterative algorithm canprovide very accurate solutions to the optimization problem as-sociated with the QoS mapping. Figs. 2 and 3 show that the op-timal solutions obtained through the new algorithm are virtuallythe same as those obtained through exhaustive search of all pos-sible values for . In contrast, the categorization QoS mappingalgorithm [6] provides coarse-grained solutions, which deviatesignificantly from the true solutions. These figures also showthat as the price per packet constraint increases more packetsare mapped into the highest Diffserv class.

Since the new QoS mapping algorithm is very accurate, itcan minimize the loss impact more effectively than the algo-rithm proposed in [6]. The normalized minimal loss impact(achieved by dividing the actual packet loss by the packet loss

Fig. 2. Distribution of video packets among the four Diffserv classes versus theprice constraint, for the “Foreman” sequence with new packet loss impact mea-surement. The results are obtained through three different methods: an exhaus-tive search (squares), the QoS mapping algorithm of the present paper (stars),and the categorization method (triangles) of [6]. The lines are for guidance only.

Fig. 3. Same as in Fig. 2 but for “Foreman” sequence with “three factor” lossimpact measurement [6].

when all packets are transmitted on the highest loss Diffservclass) achieved by the new QoS mapping algorithm decreasessmoothly as the per packet price constraint increases. This isin contrast to Shin’s categorization optimization algorithm.Results are shown in Fig. 4(a) and 4(b) and Fig. 5(a) and 5(b).

For a given loss impact measurement, the objective qualityobtained with the new QoS mapping algorithm is better thanwith the Shin’s method, as shown in Fig. 4(c) and 4(d) andFig. 5(c) and 5(d), where the PSNR values are statistical aver-ages over 100 runs of different packet loss patterns.

In Fig. 6, we plot the average PSNR value versus the framenumber. For the “Foreman” sequence, the new QoS mapping al-gorithm is slightly better than with the categorization QoS map-ping algorithm for both the new loss impact measurement andthe three-factor method [6]. This is consistent with the fact that

1008 IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 8, NO. 5, OCTOBER 2006

Fig. 4. Normalized minimal loss impact and the average PSNR achieved withthe new QoS mapping algorithm (stars) and by the categorization algorithm (tri-angles) for “Foreman” sequence. (a) and (c) new loss impact measurement and(b) and (d) the “three-factor” measurement.

Fig. 5. Same as in Fig. 4, but for the “Mother & Daughter” sequence.

at the price constraint the all-frame averaged PSNR values, ac-cording to the data presented in Fig. 4(c) and 4(d) differ by0.06 dB and 0.17 dB in Fig. 6(a) and 6(b) respectively. However,for the “Mother & Daughter” sequence, the new QoS mappingalgorithm yields noticeably better results, and in this case theall-frame averaged PSNR values differ by 0.73 dB in Fig. 6(c)and 0.28 dB in Fig. 6(d).

We observe that the new loss impact measurement performsvery similarly to the three-factor method. Both of them can pro-vide good protection to I-frames. (The peaks in Fig. 6 occur atthe I-frames.) The reason is that both loss impact measurementsusually yield higher loss impact for the I-frames than for theP-frames, resulting in the I-frames getting higher chances to bemapped into high Diffserv classes that suffer low loss rates.

We also studied another Diffserv network model with threeaggregated traffic classes, where the loss rate in the low,

Fig. 6. PSNR versus frame number. (a) “Foreman” sequence using the newloss impact measurement, (b) “Foreman” sequence using the “three-factor”loss impact measurement, (c) “Mother & Daughter” with new loss impact, and(d) “Mother & Daughter” with “three factor” loss impact. The solid lines arefor new QoS mapping algorithm, the dashed lines are for the categorizationalgorithm of [6]. We also plot the PSNR of the nonerrored bit stream (dottedlines) for reference.

medium, and high priority classes is 10%, 2.5%, and 0%,respectively. We found that the simulation results were qual-itatively the same as described above. In particular, the newoptimal QoS mapping algorithm can minimize the total lossimpact more effectively, and produce better objective videoquality than the categorization optimal QoS mapping algorithm.Results for this case have been reported in [12].

A series of subjective tests was also undertaken to assess theperformance of the new technique. These were based on a mod-ified form of the Double Stimulus Continuous Quality Scale(DSCQS) method [15]. As the sequences to be measured aresignificantly degraded, it is not appropriate to use the originalsequence (i.e., no compression or data loss) as a reference as isused in the standard form of this test. Instead, we use the newQoS mapping algorithm as the reference technique that is pairedwith the categorization QoS mapping and the two methods arethen compared directly. Viewers were asked to score the rela-tive subjective quality of each pair of sequences on a continuousscale divided into five divisions marked excellent, good, fair,poor and bad as a guide. The difference between the midpointof adjoining division therefore corresponds to a 25% differencein the result.

Three sequences were used for the tests, “Mother &Daughter” and “Foreman” as previously described and addi-tionally the “Stefan” sequence coded at 640 kbps. Since we aredealing with random packet loss, there is no single sequencethat uniquely corresponds to a given set of network conditions.For each test 20 different packet loss patterns were generatedand the pattern whose corresponding decoded sequence hasthe closest PSNR value to the mean for the set of 20 was thenused. Fig. 7 shows the results of the subjective tests that wereconducted with 16 nonexpert viewers. The error bars showthe 95% confidence levels for each test. The figure shows that

ZHANG et al.: EFFICIENT STREAMING PACKET VIDEO 1009

Fig. 7. Subjective test results. The vertical scale shows the difference between the subjective quality measured for the categorization QoS mapping algorithm andthe new QoS mapping which is the reference condition. Positive values on this scale denote worse subjective quality than the reference Beneath each error bar isthe average price per packet and the objective quality (PSNR) difference between the two techniques.

the new QoS mapping algorithm consistently outperforms thecategorization method both in the subjective quality (by abouthalf a division on the DSCQS scale), as well as in objectivequality.

V. CONCLUSION

We have addressed two important issues in streaming videoover differentiated services networks. First, the loss impact ofa video packet has been analyzed and a simple method to mea-sure the video quality degradation due to packet loss has beenproposed. We have shown that packet prioritization based onthe proposed loss impact measure achieves comparable perfor-mance to the method in [6] with lower computational com-plexity.

We have also studied the optimal QoS mapping from a videopacket sequence to a set of Diffserv classes. The results shownindicate that performance of the new algorithm is nearly iden-tical to an exhaustive search (computationally infeasible for realapplications) for the best allocation between classes. This isbacked up by subjective test results that show the difference be-tween the new algorithm and the method in [6] can be detectedby nonexpert viewers.

REFERENCES

[1] R. Braden, D. Clark, and S. Shenker, “,” Integrated Services in the In-ternet Architecture: An Overview Jun. 1994, RFC 1633.

[2] S. Blake, D. Black, M. Carlson, E. Davies, Z. Wang, and W. Weiss, AnArchitecture for Differentiated Services Dec. 1998, RFC 2475.

[3] C. Dovrolis and P. Ramanathan, “A case for relative differentiated ser-vices and proportional differentiation model,” IEEE Network, vol. 13,pp. 2–10, Sep./Oct. 1999.

[4] J. Heinanen, F. Baker, J. Weiss, and W. Wroclawski, Assured For-warding PHB Group Jun. 1999, RFC 2597.

[5] Y. T. How, D. Wu, B. Li, T. Hamada, I. Ahmad, and H. J. Chao, “A dif-ferentiated services architecture for multimedia streaming in the nextgeneration internet,” Comput. Netw., no. 32, pp. 185–209, 2000.

[6] J. Shin, J. G. Kim, and C.-C. J. Kuo, “Quality-of-service mappingmechanism for packet video in differentiated services network,” IEEETrans. Multimedia, vol. 3, no. 2, pp. 219–231, 2001.

[7] J. G. Kim, J. Kim, J. Shin, and C.-C. J. Kuo, “Coordinated packet levelprotection employing corruption model for robust video transmission,”in Proc. VCIP, San Jose, CA, Jan. 2001, pp. 410–421.

[8] J. Shin, J. G. Kim, J. W. Kim, and C.-C. J. Kuo, “Dynamic QoS map-ping for streaming video in relative service differentiation networks,”Eur. Trans. Telecommun., vol. 12, no. 3, pp. 217–230, May/Jun. 2001.

[9] H.-R. Shao, W. Zhu, and Y.-Q. Zhang, “User-aware object-based videotransmission over the next generation internet,” Signal Process.: ImageCommun., vol. 16, pp. 763–784, 2001.

[10] T. Ahmed, G. Buridant, and A. Mehaoua, “Encapsulation and markingof MPEG-4 video over IP differentiated services,” in Proc. IEEE Symp.Computers and Communications, Tunisia, 2001, pp. 346–352.

[11] H. Yu, D. Makrakis, and O. Barbosa, “Experimental evaluation ofMPEG-2 video over differentiated services IP networks,” in Proc.IEEE Pacific Rim Conf. Communications, Computers and SignalProcessing, 2001, pp. 369–472.

[12] F. Zhang, M. R. Pickering, M. R. Frater, and J. F. Arnold, “StreamingMPEG-4 video over differentiated services networks,” in Proc. 1stWorkshop on the Internet, Telecommunications and Signal Processing,Wollongong, Australia, Dec. 2002, pp. 100–105.

[13] “Optimal quality-of-service mapping for streaming video over differ-entiated services networks,” in Proc. ICASSP, Hong Kong, Apr. 2003,vol. 5, pp. 744–747.

[14] A. R. Reibman, V. A. Vaishampayan, and Y. Sermadevi, “Quality mon-itoring of video over a packet network,” IEEE Trans. Multimedia, vol.6, no. 2, pp. 327–334, Apr. 2004.

[15] Methodology for the Subjective Assessment of the Quality of Televi-sion Pictures 2000, ITU Rec. ITU-R BT.500-10.

Fei Zhang was born in China in 1964. He receivedhis B.Sc. degree in mathematics from the Universityof Science and Technology of China in 1985 and thePh.D. degree in theoretical and mathematical physicsfrom the University of Madrid, Spain, in 1992.

During 1993-2002, he was a Research Fellowand Senior Research Fellow with the AustralianNational University, The National University ofSingapore, and the University College AustralianDefence Force Academy, The University of NewSouth Wales, Canberra, Australia, and he published

over 40 papers in the broad field of computational science. After workingas a Software Engineer with CSIRO, Australia, for two-and-a-half years, hejoined Geoscience Australia in July 2005 and is currently a System Design andDevelopment Analyst for remote sensing applications.

1010 IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 8, NO. 5, OCTOBER 2006

James Macnicol (M’03) received the B.E. degreein systems engineering from the Australian NationalUniversity, Canberra, in 1998 and the Ph.D. degreefrom University College, University of New SouthWales, Sydney, Australia, in 2003.

Since 2002, he was a Research Associate with theSchool of Information Technology and ElectricalEngineering, University College Australian DefenceForce Academy, The University of New SouthWales, Canberra, Australia. His research interestsinclude video coding and associated issues relating

to transmission of video over unreliable channels. He passed away in 2004.

Mark R. Pickering (M’96) received the B.E. degreefrom Capricornia Institute of Advanced Education,Rockhampton, Australia, in 1988, and the M. E. andPh. D. degrees in electrical engineering from theUniversity of New South Wales, Sydney, Australia,in 1991 and 1995, respectively.

He is currently a Senior Lecturer in the School ofInformation Technology and Electrical Engineering,Australian Defence Force Academy, Canberra,Australia. His research interests include videoand audio coding, data compression, information

security, data networks and error-resilient data transmission.

Michael R. Frater (S’89–M’91) received the B.Sc.and B.E. degrees from the University of Sydney,Sydney, Australia, in 1986 and 1988 respectively,the Ph.D. degree from the Australian NationalUniversity, Canberra, Australia, in 1991, and theM.H.Ed. degree from the University of New SouthWales, Sydney, in 1996.

Since 1991, he has been with the University ofNew South Wales at the Australian Defence Forceacademy, Canberra, where he is currently an Asso-ciate Professor. His research interests lie in the fields

of audio-visual and multimedia communications and communication systems.Prof. Frater has served as an Associate Editor of the IEEE TRANSACTIONS ON

IMAGE PROCESSING.

John F. Arnold (S’77–M’85–SM’96) received theB.E. and M.Eng.Sc. degrees from the Universityof Melbourne, Australia, in 1976 and 1979, respec-tively, and the Ph.D. degree from the University ofNew South Wales (UNSW), Sydney, Australia, in1984.

Since 1978, he has been with the School ofElectrical Engineering, UNSW, initially at the RoyalMilitary College, Duntroon, and more recently at theAustralian Defence Force Academy, Canberra. Heis currently a Professor of electrical engineering and

Head of the School of Information Technology & Electrical Engineering. Hisresearch interests lie in the fields of video coding, error resilience of compresseddigital video, and coding of remotely sensed data. He has published widely inthese areas. Prof. Arnold is an Associate Editor of the IEEE TRANSACTIONS ON

CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY.