Video Transcoding: An Overview of Various Techniques and Research Issues IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 7, NO. 5, OCTOBER 2005 Ishfaq Ahmad, Senior

Video Transcoding: An Overview of Video Transcoding: An Overview of Various Techniques and Research IssuesVarious Techniques and Research Issues

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 7, NO. 5, OCTOBER 2005IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 7, NO. 5, OCTOBER 2005Ishfaq Ahmad, Senior Member, IEEE, Xiaohui Wei, Student Member, IEEE, Yu Sun, Ishfaq Ahmad, Senior Member, IEEE, Xiaohui Wei, Student Member, IEEE, Yu Sun,

Student Member, IEEE, and Ya-Qin Zhang, Fellow, IEEEStudent Member, IEEE, and Ya-Qin Zhang, Fellow, IEEE

2006/09/192006/09/19S.K.ChangS.K.Chang

OutlineOutline

Video Transcoding IntroductionVideo Transcoding Introduction Video Transcoding architectureVideo Transcoding architecture Homogeneous video transcodingHomogeneous video transcoding Heterogeneous video transcodingHeterogeneous video transcoding Related research issuesRelated research issues Conclusion Conclusion

Video Transcoding IntroductionVideo Transcoding Introduction One of the fundamental challenges in deploying One of the fundamental challenges in deploying

multimedia systems is to deliver smooth and multimedia systems is to deliver smooth and uninterruptible flow of audio-visual information, uninterruptible flow of audio-visual information, anytime and anywhere. anytime and anywhere. It may consist of various devices interconnected via It may consist of various devices interconnected via

heterogeneous wireline and wireless networks. heterogeneous wireline and wireless networks. multimedia content originally authored and compressed multimedia content originally authored and compressed

with a certain format may need bit rate adjustment / format with a certain format may need bit rate adjustment / format conversion to allow access by receiving devices with conversion to allow access by receiving devices with diverse capabilities.diverse capabilities.

A transcoding mechanism is required to make the A transcoding mechanism is required to make the content adaptive to the capabilities of diverse content adaptive to the capabilities of diverse networks and client devices. networks and client devices.

Video Transcoding IntroductionVideo Transcoding Introduction

Scalable coding is another approach to enable bit-rate Scalable coding is another approach to enable bit-rate adjustment. Traditional scalability in video adjustment. Traditional scalability in video compression can be of three types: SNR scalability, compression can be of three types: SNR scalability, spatial scalability, and temporal scalability. To spatial scalability, and temporal scalability. To achieve different levels of video quality.achieve different levels of video quality. scalable coding is inflex1ible since the number of different scalable coding is inflex1ible since the number of different

predefined layers is limitedpredefined layers is limited the bit-rate of the target video can not be reduced lower the bit-rate of the target video can not be reduced lower

than the bit-rate of the base layer.than the bit-rate of the base layer.



Video Transcoding architectureVideo Transcoding architecture

TranscoderTranscoder

Open-Loop Transcoder V.S. Closed-Loop TranscoderOpen-Loop Transcoder V.S. Closed-Loop Transcoder Feedback loopFeedback loop



Drift errorDrift error



Spatial-Domain Video TranscodingSpatial-Domain Video Transcoding performs dynamic bit-rate adaptation via the rate-performs dynamic bit-rate adaptation via the rate-

control at the encoder side. control at the encoder side. is flexible since the decoder-loop and the encoder-is flexible since the decoder-loop and the encoder-

loop can be totally independent of each other loop can be totally independent of each other is drift-free, is drift-free, computational complexity is high for real-time computational complexity is high for real-time

applications.applications.



Spatial-Domain Video TranscodingSpatial-Domain Video Transcoding When transcoding without spatial/temporal When transcoding without spatial/temporal

resolution reduction, the SDTA architecture can be resolution reduction, the SDTA architecture can be further simplified in which only one feedback loop further simplified in which only one feedback loop is employed.is employed.



MV reuseMV reuse Motion estimation in video encoding accounts for Motion estimation in video encoding accounts for

60% ~ 70% of the encoder computation.60% ~ 70% of the encoder computation. Two optional functional blocks placed between the Two optional functional blocks placed between the

decoder and encoder: decoder and encoder: spatial/temporal resolution reduction (STR) Modulespatial/temporal resolution reduction (STR) Module

STR allows the source video to be transcoded to target video STR allows the source video to be transcoded to target video with different spatial/temporal resolution accordingly. with different spatial/temporal resolution accordingly.

MV composition and refinement(MVCR) ModuleMV composition and refinement(MVCR) Module MVRC is needed to adjust the MVs when STR is applied.MVRC is needed to adjust the MVs when STR is applied.

Video Transcoding architectureVideo Transcoding architecture Frequency-Domain Video TranscodingFrequency-Domain Video Transcoding

Exploiting the structural redundancy of the architecture in Exploiting the structural redundancy of the architecture in SDTA and the linearity of the DCT/IDCT, a structurally SDTA and the linearity of the DCT/IDCT, a structurally simpler but functionally equivalent frequency-domain simpler but functionally equivalent frequency-domain transcoding architecture is possibletranscoding architecture is possible

In this architecture, only VLD and inverse quantization are In this architecture, only VLD and inverse quantization are performed to get DCT value of each block in the decoder performed to get DCT value of each block in the decoder end. end.

At the encoder end, the motion compensated residue errors At the encoder end, the motion compensated residue errors are encoded through re-quantization, and VLC. The are encoded through re-quantization, and VLC. The reference frame memory in the encoder end stores the DCT reference frame memory in the encoder end stores the DCT values after inverse quantization, that are then fed to the values after inverse quantization, that are then fed to the frequency-domain MC module to reduce drift error.frequency-domain MC module to reduce drift error.

Motion compensation is performed in the frequency Motion compensation is performed in the frequency domain using a MV reusing algorithm. domain using a MV reusing algorithm.


Frequency-Domain Video TranscodingFrequency-Domain Video Transcoding An FDTA may need less computation but suffer An FDTA may need less computation but suffer

from the drift problem due to nonlinearity from the drift problem due to nonlinearity operations, which includes sub-pixel motion operations, which includes sub-pixel motion compensation, and DCT coefficients clipping compensation, and DCT coefficients clipping during MC. during MC.

FDTA is also lack flexibility and are mostly fitted FDTA is also lack flexibility and are mostly fitted for bi-rate transcoding. for bi-rate transcoding.



Hybrid-Domain Transcoding ArchitectureHybrid-Domain Transcoding Architecture

Homogeneous video transcodingHomogeneous video transcoding

Reducing Bits With Fixed ResolutionReducing Bits With Fixed Resolution Re-Quantization: Re-Quantization:

A simple technique to transcoding a video to lower bit rate is to A simple technique to transcoding a video to lower bit rate is to increase the quantization step at the encoder part increase the quantization step at the encoder part

Requantizing is a good compromise between the complexity and Requantizing is a good compromise between the complexity and reconstructed image quality,and can control the bit-rate reduction.reconstructed image quality,and can control the bit-rate reduction.

Selective Transmission: Selective Transmission: Since most of the energy is concentrated at the lower frequency Since most of the energy is concentrated at the lower frequency

band of an image, discarding (truncating) some of the higher ac band of an image, discarding (truncating) some of the higher ac frequency coefficients can preserve the picture quality, but may frequency coefficients can preserve the picture quality, but may introduce a blocking effect in the reconstructed target video.introduce a blocking effect in the reconstructed target video.


Spatial Resolution ReductionSpatial Resolution Reduction Filtering and SubsamplingFiltering and Subsampling Pixel AveragingPixel Averaging

every m m pixels are represented by a single pixel of every m m pixels are represented by a single pixel of their average value. their average value.

Discarding High Order DCT CoefficientsDiscarding High Order DCT Coefficients DCT decimation delivers better quality for image down-DCT decimation delivers better quality for image down-

sampling over filtering or pixel-averaging, but for large sampling over filtering or pixel-averaging, but for large bit rate reduction greater than 25%,this method bit rate reduction greater than 25%,this method produces poor-quality blocky pictures.produces poor-quality blocky pictures.

Homogeneous video transcodingHomogeneous video transcoding Spatial Resolution ReductionSpatial Resolution Reduction

MV Composition and Refinement:MV Composition and Refinement: RandomRandom MeanMean

This technique may yield poor results if the magnitude of one of the input MVs This technique may yield poor results if the magnitude of one of the input MVs is significantly larger than the rest.is significantly larger than the rest.

Weighted AverageWeighted Average each MV is weighted by the spatial activity of the perspective prediction error. each MV is weighted by the spatial activity of the perspective prediction error.

This method is prone to noise in candidate MVs and may bias the MV when This method is prone to noise in candidate MVs and may bias the MV when original MVs are aimed in various directions.original MVs are aimed in various directions.

Weighted MedianWeighted Median Computing the Euclidean distances between each MV. This method yields Computing the Euclidean distances between each MV. This method yields

good performance, but requires substantial computation in determining the good performance, but requires substantial computation in determining the median MV.median MV.

DCmaxDCmax This method takes a little more computation than the Mean, but yields better This method takes a little more computation than the Mean, but yields better

performance than the Mean and the WA.performance than the Mean and the WA.


Spatial Resolution ReductionSpatial Resolution Reduction MV refinements techniquesMV refinements techniques

At the encoder end are proposed. At the encoder end are proposed. Since the passed MVs will almost be the same as the Since the passed MVs will almost be the same as the

recalculated ones, we can refine them to get more recalculated ones, we can refine them to get more appropriate values. The refinement can be done in a appropriate values. The refinement can be done in a small search window around the passed MV. small search window around the passed MV.

Homogeneous video transcodingHomogeneous video transcoding Spatial Resolution ReductionSpatial Resolution Reduction

MB Coding Mode Decision:MB Coding Mode Decision: The high-quality original bitstream are not optimum fore-encoding at the The high-quality original bitstream are not optimum fore-encoding at the

reduced rate in rate reduction by requantizing.reduced rate in rate reduction by requantizing. SolutionSolution

re-evalate the macro block type at the encoder of the transcoderre-evalate the macro block type at the encoder of the transcoder Bjork’s methodBjork’s method

a) If it was coded as INTRA (at the transmitter) again code it in INTRA.a) If it was coded as INTRA (at the transmitter) again code it in INTRA. b) If it was coded as SKIPPED again code it as SKIPPED.b) If it was coded as SKIPPED again code it as SKIPPED. c) If it was coded in INTER, check to see if all coefficients are zero and c) If it was coded in INTER, check to see if all coefficients are zero and

if they are coded as SKIPPED, else check again whether the macro if they are coded as SKIPPED, else check again whether the macro block has to be coded in INTRA or INTER mode.block has to be coded in INTRA or INTER mode.

Merge methodMerge method 1) If there exists at least one INTRA type among the four MBs then pass 1) If there exists at least one INTRA type among the four MBs then pass

it as INTRA; pass as INTER type if there is no INTRA MB and at least it as INTRA; pass as INTER type if there is no INTRA MB and at least one INTER MB; pass as SKIP if all MBs are of the SKIP type.one INTER MB; pass as SKIP if all MBs are of the SKIP type.

2) Re-evaluate the MB types in the encoder.2) Re-evaluate the MB types in the encoder.


Temporal Resolution ReductionTemporal Resolution Reduction Reduction in frame rate may save bits that can be Reduction in frame rate may save bits that can be

used in the remaining frames to maintain used in the remaining frames to maintain acceptable overall picture qualityacceptable overall picture quality

With dropped frames, the incoming MVs are not With dropped frames, the incoming MVs are not valid because they point to the frames that do not valid because they point to the frames that do not exist in the transcoded bit-stream.exist in the transcoded bit-stream.

Homogeneous video transcodingHomogeneous video transcoding Temporal Resolution ReductionTemporal Resolution Reduction

Bilinear Interpolation:Bilinear Interpolation: Forward Dominant Vector SelectionForward Dominant Vector Selection

Telescopic Vector CompositionTelescopic Vector Composition accumulates all the MVs of the corresponding macro blocks of the dropped accumulates all the MVs of the corresponding macro blocks of the dropped

frames and add each resultant composed MV to its correspondence in the frames and add each resultant composed MV to its correspondence in the current frame. current frame.

Activity-Dominant Vector SelectionActivity-Dominant Vector Selection Utilizes the activity of the macro block to decide the choice of the MV. The Utilizes the activity of the macro block to decide the choice of the MV. The

activity is represented by counting the number of nonzero quantized DCT activity is represented by counting the number of nonzero quantized DCT coefficients or other statistics. These quantities are proportional to the spatial-coefficients or other statistics. These quantities are proportional to the spatial-activity measurement. The higher the activity of the macro block, the more activity measurement. The higher the activity of the macro block, the more significant will be the motion of the macro block. The computation for significant will be the motion of the macro block. The computation for counting the nonzero coefficients is very little.counting the nonzero coefficients is very little.


Transcoding Between Multiple and Single LayersTranscoding Between Multiple and Single Layers

Heterogeneous video transcodingHeterogeneous video transcoding

Main IssuesMain Issues syntax conversion syntax conversion

module, and may module, and may change the picture type, change the picture type, picture resolution, picture resolution, directionality of MVs, directionality of MVs, and picture rate. and picture rate.

Heterogeneous video transcodingHeterogeneous video transcoding

Generic Heterogeneous TranscoderGeneric Heterogeneous Transcoder Syntax conversion (SC) is needed to convert the syntax of Syntax conversion (SC) is needed to convert the syntax of

source video to that of the target video. source video to that of the target video.

Related research issuesRelated research issues

Rate Control in TranscodingRate Control in Transcoding Rate control for transcoding a pre-compressed bit stream Rate control for transcoding a pre-compressed bit stream

may exploit certain information extracted from compressed may exploit certain information extracted from compressed bit streams to assist in bit-rate regulation. This information bit streams to assist in bit-rate regulation. This information can be motion estimation, input bit rate/output bit rate, can be motion estimation, input bit rate/output bit rate, INTRA/INTER mode decision, and picture complexityINTRA/INTER mode decision, and picture complexity

A Lagrangia-based rate-distortion optimization technique A Lagrangia-based rate-distortion optimization technique has been exploited for bit allocation during transcoding, but has been exploited for bit allocation during transcoding, but it is suitable for nonreal-time application due to high it is suitable for nonreal-time application due to high computational complexity.computational complexity.

Related research issuesRelated research issues

Error-Resilient Transcoding for Video Over Wireless Error-Resilient Transcoding for Video Over Wireless ChannelChannel An error-resilient transcoder can improve video quality in An error-resilient transcoder can improve video quality in

the presence of errors while maintaining the input bit rate the presence of errors while maintaining the input bit rate over wireless channels. over wireless channels.

Reyes describe a method to maintain quality for video Reyes describe a method to maintain quality for video transcoding for wireless channels which is based on transcoding for wireless channels which is based on analytical models that characterize how corruption analytical models that characterize how corruption propagates in a video and subjected to bit errors. propagates in a video and subjected to bit errors.

Dogan used adaptive intra refresh and feedback control Dogan used adaptive intra refresh and feedback control signaling methods to improve the error resilience of signaling methods to improve the error resilience of compressed video in the transcoding operation. compressed video in the transcoding operation.

Related research issuesRelated research issues Logo Insertion Scheme in Video TranscodingLogo Insertion Scheme in Video Transcoding

Logo insertion to provide pretLogo insertion to provide pret Issue: lower the effect of logoIssue: lower the effect of logo

Object-Based TranscodingObject-Based Transcoding Object importance can be taken into accountObject importance can be taken into account

Transcoding to H.264Transcoding to H.264 H.264 is different from the previous video compression H.264 is different from the previous video compression

standards. The syntax and the algorithms used in H.264 are standards. The syntax and the algorithms used in H.264 are so different that transcoding from traditional DCT-based so different that transcoding from traditional DCT-based standards to H.264 will face many difficulties, especially to standards to H.264 will face many difficulties, especially to perform transcoding in the frequency domain. perform transcoding in the frequency domain.

4X4 Integer transformation V.S. 8X8 DCT4X4 Integer transformation V.S. 8X8 DCT Prediction block structures and MV predictionPrediction block structures and MV prediction

ConclusionConclusion Video transcoding is a core technology for providing universal Video transcoding is a core technology for providing universal

multimedia access by the users with different access links and multimedia access by the users with different access links and devices. This paper reviewed several existing video devices. This paper reviewed several existing video transcoding techniques which provide trade off between the transcoding techniques which provide trade off between the computational complexity and reconstructed video quality. computational complexity and reconstructed video quality. SDTAs provide the best video quality but with more complexitySDTAs provide the best video quality but with more complexity FDTAs provide a bit lower quality but with lower complexity. FDTAs provide a bit lower quality but with lower complexity. HDTAs take advantages of both architectures to provide a trade off HDTAs take advantages of both architectures to provide a trade off

Object-based transcoding architectures and techniques offer Object-based transcoding architectures and techniques offer important research directions. important research directions.

H.264 are very different from that of in the previous H.264 are very different from that of in the previous traditional video compression standards. H.264 related traditional video compression standards. H.264 related transcoding would become a more challenge issue in the transcoding would become a more challenge issue in the future research of video transcoding.future research of video transcoding.

Documents

Video Transcoding: An Overview of Various Techniques and Research Issues IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 7, NO. 5, OCTOBER 2005 Ishfaq Ahmad, Senior