A comparative study of adaptive degradation of video quality over

the internetShafquat Mallik

Introduction• Establishing need for graceful degradation• Video Streaming

– Requirements and nature of internet• Various Techniques

– Error Control Mechanism (None requires QoS aware network)• Synchronous:

– Uses time. – Checksum appended– Receiver sends an ack.

• Asynchronous – Uses Space– FEC– No feedback required

– Congestion Control• Receiver Based

– Data divided into layers– Receiver decides how many layers it can join.– QoS aware nw can drop low priority layers

• Sender Based– Feedback mechanism is used from all receivers

Initial Work

• PET

• ULP

• FEC

• Layering

Perceived Quality and bandwidth characterization of layered MPEG-2

video encoding.

• Various layering mechanisms are considered to show how layers should be created and prioritized to maximize the perceived quality over a given range of network conditions.

Various Layering Schemes

• Data Partitioning– Divides the encoded bit stream into two or more

layers by allocating the data in bit stream to various layers.

– PBP are introduced to specify the grouping of DCT coefficients into the layers.

– Easiest to implement.• SNR Scalability

– Encodes the video using a quantizer scale Qb to generate the base layer and then using the difference between original video and base layer using a quantizer scale Qe to generate the enhancement layer.

Approach of the paper

• All possible layering structures were considered and out of them the structure that offered best quality bandwidth trade off was selected.

Results

2 Layer DP with random packet loss in enhancement layer

Packet Loss Resilient Internet Video Streaming[1]

• This scheme uses a scalable video coder combined with unequal error protection applied across packets.

• Scalable coder produces a bit stream, decodable at different bit rates.

• Markov model with two states is used to approximate the bursty packet loss behavior – State G (Good) packets are received correctly and timely– State B (Bad) Packets are lost either due to congestion or

delayed arrival. – Average Loss probability PB = Pr(B) = PGB /(PGB+ PBG)– Average Burst Length LB = 1/PBG

• FEC scheme uses Reed-Solomon code across packets.– At the sender copies of sent information is stored until k packet

have been sent. – Then n-k redundant packets are calculated and transmitted.– These n packets are called block of packets (BOP).– To maintain a constant data rate on the channel the source data

rate must be reduced by the factor k/n• At receiver all information can be reconstructed from any

subset of at least k correctly received packets• Decoding process can start as soon as any k packets of

a BOP are received.• Residual Block Error:

– Probability that a BOP cannot be reconstructed. i.e. more than (n-k) packets are lost within a BOP.

– It is summation of probability of m lost packets within a block of n packets P(m,n).

– P(m,n) determines the performance of FEC scheme.

FEC

Scalable Video Coding with unequal error protection

• Given average packet loss rate and average burst length a video transmission scheme can be designed to guarantee a certain BER.

• Two layer scalable video coder is used that uses H.263 base layer encoding and an enhancement layer.

• Both layers are packetized into one BOP along with redundancy.

• For error concealment if less than k packets are received, for enhancement layer missing image parts are replaced by spatially interpolated image content from the base layer.

• If packet loss occurs in base layer, enhancement layer is switched off.

PSNR Estimation

• PSNR is used for quality estimation.

• Probability that layer l is displayed is

• Probability that base layer is also affected by packet loss is PL . let PSNRl be the average PSNR that layer l is displayed over the whole time period encoded in one BOP. Then the average PSNR resulting from switch off strategy is

Simulation Results

• Input sequence are foreman and silent voice encoded at 30 fps.

• 15% redundancy was allowed.• Proposed scheme achieves graceful

degradation.

Unequal Loss protection: Graceful degradation of Image Quality over Packet Erasure Channels Through

Forward Error Correction[2]

• Unequal amount of FEC is applied to progressive data to provide graceful degradation in case of packet loss.

Formalizing the Framework• Message M (a sequence of data)• mi number of data bytes assigned to stream I• fi = N – mi number of FEC bytes assigned to stream i.• Redundancy assignment is an L dimension FEC vector whose entries are

length of FEC assigned to each stream. – ƒ = (ƒ1, ƒ2,…. ƒL)

• For a given ƒ M is divided into Mi(ƒ) fragments and define it to be sequence of data bytes in the ith stream.

• Prefix of M containing the first j fragments for redundancy vector ƒ as – M(j, ƒ) = M1(ƒ) M1(ƒ) …Mj(ƒ)

• Incremental PSNR of stream i is– gi = PSNR [ M (i, ƒ) ] – PSNR [ M (i-1, ƒ)]

• The quantity c(ƒi) is the probability that receiver can decode stream I• Expected PSNR would be

– G(ƒ) = Σ c(ƒi) gi(ƒ)• A value of ƒ that maximize G(ƒ) subject to packet loss estimate Pn is

required.

Algorithm For Solving the ULP problem

• Since the globally optimal solution is computationally prohibitive for a useful amount of data, a local search hill algorithm is used that makes assumptions about data but is computationally tractable.

• Initialize each stream with data bytes only such that mi = N and ƒi =0,1,2,3…L

• In each iteration algorithm examines a number of possible assignments equal to 2QL, where Q is the maximum number of FEC bytes that can be added or subtracted to a stream in one iteration. And L is the number of streams.

• G(ƒ) is determined after adding or subtracting 1 to Q bytes of FEC data to each stream. ƒ corresponding to the highest G(ƒ) is chosen.

• Repeat the search until no more value improves the expected PSNR.

Results• Standard 512*512 and 256*256 gray scale image compressed with

SPHIT is used.• Channel loss model is exponential PMF with a mean loss rate of

20%.• ULP and ELP were compared. • ULP and ELP gave an expected PSNR of 29.42 and 28.94

respectively.• Under good channel conditions ULP performs better than ELP.• ULP degrades gracefully whereas ELP has a sharp transition at loss

rate near 51%• ULP out performs ELP 85% of the time.• System provides progressive transmission once a number of

packets equal to number of data bytes in packet stream 1 are received decoding of image can begin and as when additional packets are arrived image can improve.

Efficient partitioning of unequal error protected MPEG video streams for

multiple channel transmissions• Efficiently partitioning forward error protected,

pre-encoded video data for transmission over multiple channels.

• An algorithm is proposed which efficiently partitions FEC-protected video data across multiple channels to – minimize the bandwidth consumption and when the

aggregate bandwidth is sufficient to transmit the video or

– maximize the decodable frame rate when aggregate bandwidth is limited.

Transmission Model

• Multiple channels are used to provide a logical channel with sufficient or nearly sufficient bandwidth.

• Assume there are N channels and each is parameterized by BW, Delay (T), Packet loss rate (PLR)

• FEC across packets is applied to each coded frame to recover lost frames.

• Each channel is protected at a different level because of different PLR

• Assume the channel conditions vary in order of TG

seconds and interval of channel estimates is of multiple GOP durations.

Optimal Partitioning Problem

• A GOP consisting of Q frames is to be transmitted over N channels with different bandwidths, PLRs, and latencies.

• Let– S is the set of all the frames in a GOP.– Si(n) set of frames allocated to channel i.– Si(n)∩ Sj(n) = Φ if i <> j– S1(n) U S2(n)…. SN(n) = S– F(Aj) is the non-FEC coded data size of frame Aj of type A.

Redundancy is added in the fractional amount of factor Ri(A,n) – Total amount of data to be allocated to a channel is

• Ei = ∑ ( 1 + Ri ( A,n ) ) . F ( A j )

Achieving Minimum Bandwidth

• Assume channels are sorted in ascending order of reliability.

• Assume BW of channel 1 is unconstrained• More reliable channels 2 thru N are unable to

transmit the entire FEC coded GOP within TG seconds.

• Goal is to minimize the required BW of channel 1 with the constraints of channel 2 to N bandwidths.

• Let the solution be partition S = Ś1UŚ2U…..ŚN

that gives min E1 subject to Ei ≤ BWi,. TG

Achieving Maximum Frame Rate

• Objective is to transmit as many frames as possible in one GOP within limited bandwidth-time product.

• Frames in GOP are reordered based on their encoding dependencies.

• Reordering pattern can be determined prior to the start of transmission for fixed GOP patterns.

• Optimizing problem is to maximize the number q of transmitted frames. such that the Ei ≤ BWi,. TG

Pruned Tree Search Algorithm

• Search space is O(NQ) for the solution to the optimization problems.

• Tree depth corresponds to number of frames and branches represent channel assignments. Constraints are associated with each node.

• Impossible situations are pruned• If channel estimates are inaccurate greedily

either frames are dropped or added at the end.

Experiments and Results

• Minimizing BW consumption:– Experiments show that formulation achieves a

high BW utilization of channels 2 to 3 and PTS finds the optimal solutions as found by a full search.

– The algorithm is slowest when equal bandwidth is assigned to channels because pruned tree has more balanced branches.

– Computation time reduces by a factor of 5-10 when channel BW are very unequal.

Maximizing Frame Rate• Comparison with the Greedy Partitioning:

– With total available bandwidth ranging from 70% to 150% of video average bit rate and three channels of equal bandwidth.

– Proposed system outperforms greedy allocation with maximum difference from 0.64 to 2.12 frames/GOP and minimum range from -0.07 to 0.64 frames/GOP.

• Effect of Differing Video Content:– Various video content generates different patterns of frame size

traces.

• Effect of Channel Parameter Mismatch:– Performance of the transmission model depends on the

accuracy of the channel parameters provided.– At 130% estimated normalized bandwidth the average frame

rate drops by 6-7 frames/GOP when actual bandwidth is -15% lower than the estimates and increase by 1-2 frames/GOP when the actual value is +5% higher.

An adaptive Protection Framework for MPEG Base Layer Video

Streaming Over Internet

• After reinvestigating the PET an adaptive optimization based protection assignment framework for streaming MPEG base layer under a bandwidth constraint.

• An assignment is a vector of X of dimension m, • X= (x1, x2, … xm), for all i 1≤xi≤N for message fragment I• Algorithm calculates the value xi which determines the

amount of protection associated with fragment• Amount of protection used for fragment is ri*N/xi

• For x1= r1/L all parity data is assigned to protect the first frame only.

• General problem is to find an assignment that optimizes expected received quality under a given rate budget R, satisfying the constraint x1≤x2≤x3≤….xm

PET ULP system• Each message is an independent video subsequence representing

the boundary of error propagation.• N is the number of packets used to carry the encoded GOP data

and FEC parity data.• L is the size of each individual packet.• A message is partitioned into m partitions each of size ri

• A priority is assigned to each message and message is protected according to its priority.

• Given algorithm calculates xi which is amount of protection associated with each fragment i.

• Amount of protection used for fragment is (ri*N/xi)• Problem is to find an assignment that optimizes expected received

quality under the given rate budget R, satisfying the constraint – x2≤ x3≤ x4≤ x5≤….. xm

Definition of Utility-cost Function• Defined for each independent GOP.

• Cost for achieving quality at a given level i is the cumulative bit-rate required to correctly decode the message from the lowest level to the ith level.

• Utility at that level is the quality gained from correctly decoding the message segment up to that level.

• Quality is defined as average PSNR value computed over the entire GOP.

• Utility cost function is computed in two steps.– Designate number of levels and the frame loss compensation pattern at

each level.

– A utility is computed according to the designated frame_loss_compensation pattern for each level in terms of PSNR considering

• Order of frame importance. (I>P>B)• Dependency between frames. • Importance of data. (Motion vector> Texture data…)

Adaptive Optimization Assignment Framework

• The system is designed to operate in real time in response to feedback of conditions of the transmission channels.

• Inputs to module are N number of packets to be used by ULP system, channel error probability distribution, bandwidth budget for a particular GOP, Utility cost function for the GOP

• The optimization algorithm searches for the assignment that maximizes E[U]

• When number of levels is small an exhaustive search is sufficient.

• For large number of levels a pruning algorithm is used that prunes off infeasible regions, then exhaustively searches for best assignment in feasible region.

Experiments & Results

• Experiment 1:– Comparison of PET

with optimized PET• At six different mean

packet loss rate scenarios: 0.05, 0.1, 0.15, 0.2, 0.25, and 0.3.

• Optimized PET outperforms original by over 1 DB.

• Experiment 2:– Varying bandwidth to

evaluate adaptation.– Redundancy available

is 10%, 20%, 30%, 40%, 50% and 60% over original data.

– Original PET has over 30%

– Optimized system outperforms original.

H.263 Standard[5]• Low bit rate DCT based, Motion compensated coding on internet

and wireless networks.• Frame structure:

– Supported formats QCIF, sub-QCIF, CIF, 4CIF, 16CIF. Frame is split into 16*16 pixel MB.

• Motion Compensation:– Inter (I) frame containing only intra macro blocks. First motion

compensated and then the difference is encoded– Intra (P) frame containing both intra and inter macro blocks. Coded

independently.• Transform and Entropy Coding

– 8*8 block is transformed using DCT. – This block is scanned into a one dimensional array ordered from low to

high frequency. – Run length coding is applied.

Unequal Loss Protection (ULP)

• ULP framework operates by applying unequal amount of FEC to data that are compressed with an unmodified embedded algorithm and transmitted over lossy packet networks.

• Based on priority encoding transmission that assigns unequal amount of FEC to data sent over lossy packet networks.

• The packet is divided into L streams and each contains symbols from each of N packets. Given stream can be decoded successfully as long as number of lost packets is less than or equal to number of FEC symbols.

• ULP algorithm operates in the form of a utility curve function. Utility is how much benefit the user is likely to enjoy by receiving and decoding the stream and Cost is number of bits used to achieve a given utility.

Applying ULP to H.263

• Several changes to original ULP system are required.– Create Independent Subsequences.

• GOP size is influenced by application of interest.

– Reorder the compressed bit stream.• First I frame is a GOP• Picture headers from all frames a GOP• MB headers from all frames in a GOP….

– Select the encoding bit rate.• Generate a family of utility-cost curves.

Results & Methodology

• Expected frame PSNR.– Results show that the expected performance of ULP

system is much better than baseline H.263 due to packet loss protection used by ULP.

• Frame PSNR at various packet loss rates:– No packet loss H.263 performs better since the

encoding is at the target bit rate. Whereas ULP uses a lower bit rate with redundancy.

– At mean loss rate (9.38%) ULP provides a better quality where H.263 has an unusable video sequence.

– At very high loss rate (40.63) ULP still provides a good quality which is only slightly degraded.

Multicasting of Adaptively-encoded MPEG4 over QoS-aware IP

networks.

Multicast congestion control for single rate (sender based) single source IP multicast sessions over a QoS aware IP network.

Multicast congestion control techniques• Single rate: (Sender based)

– Single rate is sent to all receivers.

– Scalable Feedback Control uses messages from receivers to estimate group reception

• Scalability

• Choosing group representative.

• Single slow receiver can drag down the data rate for whole group.

• Multi rate: (Receiver Based)– Generate the source data in layered format. Receiver

decides how many layers it can join.• Best effort network can drop packets in the base layer

making enhancement layers useless.

RED (Random Early Detection)

• A buffer management technique used for congestion avoidance in IP networks.

• Detect upcoming congestion by computing an average of queue size in the router.– IF queue-average < min

• THEN no action taken

– IF queue-average > min AND queue-average < max • THEN with probability drop packet

– IF queue-average > max• THEN drop packet.

Backward Explicit Congestion Notification

• A scalable mechanism provided by the routers than receivers.

• No overhead of selecting representative.• Compatible with a variety of transport layer protocols. • If Queue size is between min and max a packet is

marked and sent to receiver, and receiver marks a flag in ACK message and send it to sender

• Sender reduces the congestion window and slow start threshold. And sends a notification to receiver. – Coupled with TCP – Takes a round trip time before sender reacts.

• BECN send the notification even when queue is > max.

• MPEG 4 is used because it can encode real time video adaptively to target a certain rate.

• Encoded video can be generated in layers.

Network Model

• Network should support priority dropping of packets in time of congestion

• Provide congestion notification from router to the multicast sender.

Flow control Algorithm

• REPEAT every t seconds– IF NoFeedback message received – THEN

• increase Total_Rate• Reduce basicLayerPortion• Reduce t

– ELSEIF Feedback increasing at all priority levels– THEN

• reduce TotalRate• Reduce t

– ELSEIF Feedback increasing at all Basic layer– THEN

• Reduce basicLayerPortion• Reduce t

• END REPEAT

END to END Architecture

• Sender marks the packet with two different priorities, with basic information marked with higher priority and enhancement info with lower priority.

• While congestion is developing routers that run RED will send back BECN messages to sender with information on priority level that causes congestion.

• Based on the rate of these feedback messages the sender runs an algorithm to search for an operating point that will reduce this feedback message rate.

• Sender will try to match the sending rate for the high priority with the sending rate of slow recievers.

Simulation and Performance

Results

H.264 Standard[1]

• Introduction.– Digital Video codec noted for very high compression

(2X or more than previous standards [2])– Very recent development– Intent was to develop a video codec that can provide

good video quality at bit rates lower than previous standards, without much increase in complexity

– Another goal was the application of the standard to a wide variety of applications, and on wide variety of networks and systems. http://www.lsilogic.com/technologies/industry_standards/mpeg_based_standards_h_264.html

• Consists of two conceptually different layers– Video coding layer:

• consists of specifications of core video compression engines like motion compensation, transform coding of coefficients and entropy coding

• Layer is transport-unaware.• Highest data structure is video slice.

– Network abstraction Layer:• Responsible for the encapsulation of the coded

slice into transport entities of the network.

Error Resilience Tools: Data Partitioning

• Three different partition types are used;– Partition A, Containing header information

such as Macro block types, quantization parameters, and motion vectors.

– Partition B (Intra Partitioning) carrying intra-coded block patterns and intra coefficients.

– Partition C (Inter Partitioning) Containing only inter CBP and inter coefficients.

IEEE 802.11e

• Developed in late 2005.

• Critically important for delay sensitive applications.

• Enhances the IEEE 802.11 MAC layer.

Distributed Coordination Function (DCF)

• Before initiating the transmission each station is required to sense the medium.

• If medium is busy the transmission is deferred and a back off timer is initiated by the station.

• Once the station detects that medium is free for a duration of DCF inter frame spaces it begins to decrement the timer for as long as the channel is idle. If timer expires and channel is still free, transmission begins.

• In case of a collision, the size of CW is doubled.• All stations compete for channel access with same

priority,

Enhanced Distributed Access Channel (EDCA)

• QoS support is realized with the introduction of access categories (ACs).

• Each AC has its own transmission queue and its own set of channel access parameters.

• Service differentiation is achieved by setting different CWmin , CWmax,, arbitrary inter frame spaces (AIFS) and transmission opportunity duration limit (TXOPlimit)

• If one AC has a smaller AIFS or CWmin , CWmax, The AC traffic has a better chance of accessing the wireless medium earlier.

Approaches to Reliable Video Communication over LAN

• Asynchronous: – Divide data into packets and append a special error

check sequence. An Ack is sent for each intact packet received

• Synchronous:– e.g. FEC introduces parity packets. No feedback

channel is required. If number of erased packets is less than decoded threshold the original data can be recovered perfectly.

• Cross layer Architecture:– MAC-Centric: the application layer passes its traffic

information and requirements to MAC, which maps these partitions to different traffic categories to improve perceived video quality.

Toward an improvement of H.264 video Transmission over IEEE 802.11e through a Cross-Layer

Architechture.[3]

• Challenge: Transmission over WLAN by proposing a QoS cross layer architecture based on application layer and MAC layer features.

Cross-Layer Architecture

• VCL produces following partitions, A, B, C and an additional slice that represents instantaneous decoding refresh (IDR) picture.

• These Partitions are directed towards NAL layer.

• First slice unit transmitted contains PSC information and next contains IDR picture.

• After which any of A,B or C is transmitted.

NAL Layer

• NAL layer facilitates the delivery of H0264 VCL data to the underlying transport layer.

• At this point NAL contains three fields. Paper however focuses upon Nal_Ref_Idc (NRI)

• NRI contains two bits that that indicate the priority of the NALU payload.

MAC Layer

• IEEE 802.11e defines four access categories• With AC3 being the highest and AC) the lowest. • Based on this traffic specification it is possible to

differentiate the H.264 partitions at the MAC layer• The paper proposed a marking algorithm that uses NRI

field to map the H.264 stream to a suitable traffic class.• Each NALU arrives at the MAC layer along with a priority

value. • NALU is encapsulated into a QoS data frame, where the

traffic category identifier (TID) (in MAC header) is used to differentiate between AC[i]’s frame

Marking Algorithm

Case of NRI

11 then Qos_TID = t1

10 then Qos_TID = t2

01 then Qos_TID = t3

PSC is mapped to AC3, IDR and A are mapped to AC2 and B,C are mapped to AC1 thus saving them from best effort AC0.

• The parameter (maximum retry limit) is used to unequally protect the high priority data by increasing the value of this parameter.

• For AC3 and AC2 the maximum allowed value by 802.11 MAC layer is used.

• For AC1 and AC0 a smaller value is used to limit the MAC’s retransmission of AC1 and AC0 packets.

Simulation and Results• H.264 Foreman sequence coded at 25 frames/sec with intra period

of 50.

• Unicast video transmission.

• Each simulation runs consists of 15.1 s of simulated network life.

• Packet loss rate between IDR (with Qos Architecture), EDCA and DCF shows that DCF and EDCA the mean loss experienced was 30 to 16 percent where as IDR had none.

• Instantaneous loss rate is 2% in proposed system and 13.75 and 27% in EDCA and DCF respectively for packet A.

• Packet B and C experienced higher packet loss than EDCA and DCF because of the lower priority of AC1 reduces the transmission opportunities.

• The delay experienced by IDR and packet A is minimum

• From a dropped frames of 87 and 41 picture cannot be decoded when transmitted video in DCF and EDCA but cross layer architecture was able to decode the whole sequence.

Comparative review of studied techniques.

• 1.Perceived Quality and bandwidth characterization of layered MPEG-2 video encoding

– Requires QoS aware IP network.– Applicable to MPEG only.– Exploits layering.

• 2.Packet Loss Resilient Internet video streaming– Exploits both layering and FEC– Uses H.263 coder – Requires probability of packet loss on a network and average burst length.

• 3. Unequal Loss protection: Graceful degradation of Image Quality over Packet Erasure Channels Through Forward Error Correction.

– Applies FEC on compressed images thru SPHIT – Requires channel estimator– Progressive Data

• 4. Efficient partitioning of unequal error protected MPEG video streams for multiple channel transmission

– Channel characteristics (band width, packet loss ratio, delay) are monitored and estimated at regular intervals.

• 5. An adaptive Protection Framework for MPEG Base Layer Video Streaming Over Internet

– ULP using PET FEC– Feed back requires on condition of transmission channel.

• 6. Unequal Loss protection for H.263 compressed video. – Lossy Network.– H.263 compressed video.– Reordering of the bit stream according to relative importance.– Numerous and smaller chunks of data.

• 7. Multicasting of adaptively encoded MPEG4 over QOS aware IP networks– QoS Aware network– Multicasting– MPEG4 video over IP NWKS.– Congestion control techniques.

• 8. Toward an improvement of H.264 video transmission over Ieee 802.11 through a cross layer architechture.

– Transmission of h.264 over WLAN– Based on application layer and MAC layer,– Data partitioning at application layer – QOS mapping at MAC layer

References:• [1] Packet Loss Resilient Internet Video Streaming; B. Girod, K. Stuhlmuller, M. Link, U. Horn

• [2] Unequal Loss Protectio: Graceful Degradation of image quality over Packet Erasure Channels Through Forward Error Correction. Alexander E. Mohr, Eve A. Riskin, Richard E. Ladner.

• [3] Wikipedia http://en.wikipedia.org/wiki/H.264

• [4]http://www.lsilogic.com/technologies/industry_standards/mpeg_based_standards_h_264.html

• [5] Toward an improvement of H.264 video Transmission over IEEE 802.11e through a Cross-Layer Architechture; Adlen Ksentini, Mhamed Naimi, Abdelhak G Toward an improvement of H.264 video Transmission over IEEE 802.11e through a Cross-Layer Architechture.[3] ueroui; IEEE communication magazine January 2006

• [6] IEEE 802.11e Wireless LAN for Quality of Service; Stefan Mangold, Sunghyun Choi, Peter May, Ole Klein, Guido Hiertz, Lothar Stibor;

• [7] WIkipedia http://en.wikipedia.org/wiki/H.263

• [8] Unequal loss protection for H.263 compressed video; Justin Goshi. Alexander E. Mohr, Richard E. Ladner, Eve A. RIskin, Alan Lippman