Transcoding of an MPEG-2 bit stream to an H.264 bit stream What is Transcoding ? The operation of converting video in one format to another format. Need: Compatibility between MPEG-2 and H.264devices Applications: To adapt the bit rate of a compressed stream to the channel bandwidth, to change the spatial or temporalresolution of a compressed stream etc. Criteria considered in Heterogeneous Transcoding
Quality of the transcoded stream should be comparable to that obtained by complete decoding and re-encoding with full motion search and to that of the initial input stream. The information in the input bit stream should be re-used as much as possible to reduce multigenerational degradation. The computational cost and complexity should be kept minimal. MPEG-2 Decoder Variable Length Decoding Inverse Scan Quantization
Inverse DCT Motion Compensation Frame store Memory Decoded Pels + MPEG-2 bit stream H.264 encoder Transcoding Algorithm Intra frame coding MPEG-2 H.264 Macroblock modes supported 8x8
16x16 with 4 directional modes, 4x4 with 9 directional modes Type of intra prediction Fixed prediction of D.C. coefficient Adaptive directional prediction of4x4 or 16x6 pixel blocks Transform 8x8 DCT 4x4 Integer transform Intra Frame Transcoding
MPEG-2 bit stream H.264 bit stream - + VLC Decode Inverse Quantize IDCT Mode Decision Spatial Prediction 4x4 Integer Transform Quantize Entropy Coding Complexity of applying mode decisions in transform domain
Example: Vertical prediction Predicted block = Intra modes in H.264/AVC Directional modes for an intra 4x4 macroblock
Directional prediction Directional modes for an intra 16x16 macroblock Mode decision algorithm Why use standard deviation?
Simple metric Can be easily computed as the transform domain coefficients are already available. Post mode decision Intra Frame transcoding results Subjective Quality of Intra frames MPEG-2 Input Stream H
Subjective Quality of Intra framesMPEG-2Input Stream H.264 bit stream obtained H.264 bit stream Test clip :Akiyo by the proposed method obtained from complete Bit rate: 1 MbpsBit rate: 768 Kbpsdecoding and re-encoding Spatial resolution: 352x of the input MPEG-2bit stream Bit rate: 670 Kbps Subjective Quality of Intra frames MPEG-2 Input H
Subjective Quality of Intra framesMPEG-2Input H.264 bit stream obtained H.264 bit stream bit streamby the proposed method obtained from complete Test clip: Foreman Bit rate: 1Mbpsdecoding and re-encoding Bit rate: 1 Mbps of the input MPEG-2bit Spatial resolution: 352x stream Bit rate: 1Mbps Inter frame coding MPEG-2 H.264 MC prediction with pel accuracy
No, only pel accuracy yes MC modes 16x16 16x16,16x8,8x16, 8x8,8x4,4x8, 4x4 Multiple reference prediction no Direct modes in B frames Use of B frames as reference frames Allowed, can be selected by the user Inter frame transcoding
MPEG-2 bit stream Inverse quantise Sum residuals VLD IDCT Hierarchical mode decisions Pass parameters MV Refinement Inter prediction Rate control H.264 bit stream 4x4 Integer transform VLC/CABAC Quantise Inverse VLC/CABAC Inverse Quantise Motion compensate Store as reference frame Inter frame transcoding
Features: Motion vector extraction Motion vector refinement Motion vector reuse Hierarchical mode decision Motion vector extraction
Motion vectors can be extracted from the MPEG-2bit stream after variable length decoding. Need for motion vector refinement
Differences in the quantization parameters of the incoming bit stream and those selected may differ. When these differences are large it results in quality degradation. MPEG-2 supports certain modes in which no motion information is coded. However, since H.264 supports more fine motion estimation block sizes, a small amount of motion may result upon refinement. Re-evaluation of the decision to intra code macroblocks in a P frame. Improves accuracy of the motion vectors and helps achieve compatibility between pel MV accuracy in MPEG-2and pel MV accuracy in H.264 Need for motion vector refinement
Compensates for field coding to frame coding changes and vice versa Motion vector refinement
MPEG-2 motion vectors are refined over a one pixel window i.e. dx = dy = 3 pixels , in the most recent reference frame in List 0. Half pixel and quarter pixel refinement is performed with the defined window. Search window size (dx,dy) selection
Before window size selection, different increasing window sizes were tested to verify the effect of varying the search window size on the PSNR. The graph for one such test clip Akiyo is as follows It was observed that the PSNRobtained for a one pixel window closely approximated the steady state value and using a one pixel window provided a good tradeoff between complexity and the PSNR. Motion vector reuse Hierarchical mode decisions
Coding modes are compared and selected based on the sum of absolute difference (SAD) value. In the full mode decisionmethod, every coding mode is evaluated ,the SAD value is computed and the mode with the minimum SAD value is selected as the best mode. However ,although this method would give the best results ,it is very computationally intensive. For instance, each macroblock inthe P frame would have to be evaluated for 16x16, 2 16x8, 2 8x16, 48x8, 8 8x4,8 4x8, 16 4x4intra and skip modes Hierarchical mode decision process makes use of the fact that after evaluating a mode and the next level of sub partitioned modes , ifsub partitioning does not reduce the SAD value then further sub partitioning need not be evaluated. Hierarchical mode decisions
The top down splitting approach is shown below P frame transcoding Results P frame results (see previous slide) P frame results Motion vectors in P frames MPEG-2 Input Stream H
Motion vectors in P frames MPEG-2Input Stream H.264 bit stream obtained H.264 bit stream Test clip :Akiyo by the proposed method obtained from complete Bit rate: 1 Mbps Bit rate: 768 Kbpsdecoding and re-encoding Spatial Resolution:352x of the input MPEG-2bit stream Bit rate: 670 Kbps H.264 motion vectors after transcoding MPEG-2 motion vectors H.264 motion vectors after full motion search Mode decisions in P frames MPEG-2 Input Stream H
Mode decisions in P frames MPEG-2Input Stream H.264 bit stream obtained H.264 bit stream bit stream by the proposed method obtained from complete Test clip :Akiyo Bit rate: 768 Kbpsdecoding and re Bit rate: 1 Mbps encoding of the input Spatial resolution:352x MPEG-2 bit stream Bit rate: 670 Kbps 16x16 modes in the MPEG-2 bit stream 16x16 and sub-macroblock modes in the H.264 transcoded bit stream 16x16 andsub-macroblock modes in the re-encoded H.264 bit stream B frame transcoding results B frame transcoding results B frame transcoding results Motion vectors in B frames MPEG-2 Input Stream H
Motion vectors in B frames MPEG-2Input Stream H.264 bit stream obtained H.264 bit stream Test clip :Akiyo by the proposed method obtained from complete Bit rate: 1 MbpsBit rate: 768 Kbpsdecoding and re-encoding Spatial resolution:352x of the input MPEG-2bit stream Bit rate: 670 Kbps Backward motion vectors in the H.264 transcoded bit stream Forward motion vectors in MPEG-2 Forward motion vectors in the H.264 transcoded bit stream Forward motion vectors in the H.264 transcoded bit stream Backward motion vectors in the H.264 transcoded bit stream Backward motion vectors in MPEG-2 Mode decisions in B frames MPEG-2 Input Stream H
Mode decisions in B frames MPEG-2Input Stream H.264 bit stream obtained H.264 bit stream bit stream by the proposed method obtained from complete Test clip :Akiyo Bit rate: 768 Kbpsdecoding and re-encoding Bit rate: 1 Mbps of the input MPEG-2bit Spatial resolution:352x stream Bit rate: 670 Kbps 16x16 and sub 16x16 modes in the H.264 transcoded bit stream 16x16 and sub 16x16 modes in the H.264 re-encoded bit stream 16x16 modes in the MPEG-2 bit stream B frame transcoding results Comparison of the Input MPEG-2 bit stream vs. the transcoded H
Comparison of the Input MPEG-2 bit stream vs. the transcoded H.264 bit stream The table below illustrates the comparison between the PSNR of the input MPEG-2 bit stream and the PSNR of the transcoded H.264 bit stream obtained by transcoding 35 frames at 1Mbps with the IBBPBBP GOP structure Comparison of the proposed method with the DCT domain transcoder proposed by Chang and Messerschmitt [23] The graph shown compares the proposed method with DCT domain transcoding [23] and complete decoding and re-encoding of a 1 Mbps MPEG-2 bit stream (test clip Foreman) to an H.264 bit stream with an IBBPBBP.GOP structure at a constant bit rate. Proposed method transcoded stream
Full re-encoding References [1] J. Youn and M-T. Sun , Motion Vector Refinement for high-performance transcoding, in IEEEInt. Conf. Consumer Electronics, Los Angeles, CA, Vol. 1, Issue 1, pp , March 1999. [2] J. Xin, C-W. Lin and M-T. Sun, Digital Video Transcoding , Proceedings of the IEEE, Vol. 93, pp , Jan [3] T. Wiegand et. al., Overview of the H.264/AVC Video Coding Standard, IEEE Trans. CSVT, Vol. 13, pp , July 2003. [4] A. Vetros, C. Christopoulos and H. Sun, Video transcoding architectures and techniques: an overview, IEEE Signal Processing magazine, Vol. 20,pp ,March 2003. [5] H. Kalva, Issues in H.264/MPEG-2 Video Transcoding, IEEE Consumer Communications and Networking Conf., CCNC 2004, pp , Jan 2004. [6] Information Technology-Generic coding of moving pictures and associated audio information: Video, ITU-T Rec. H.262 (2000 E). [7] B. Haskell, A. Puri and A. Netravali, Digital Video: an introduction to MPEG-2, N.Y. Chapman and Hall, International Thomson Pub., 1997. [8] G. Chen et. al., Efficient block size selection for MPEG-2 to H.264 transcoding, Proceedings of the 12th annual ACM International Conference on Multimedia, pp , Oct [9] MPEG-2 software (version 12) from MPEG software simulation group, [10] H.264 Software (JM9.5) from [11] A. Puri, X. Chen and A. Luthra, Video coding using the H.264/MPEG-4 AVC compression standard, Signal processing: Image communication, Vol. 19, pp , Oct [12] B. Shen and I. Sethi, Direct feature extraction from compressed images, SPIE: Vol Storage and Retrieval for Image Databases IV, pp , 1996. [13] Commercially available transcoders, PSP Video 9, [14] K.R. Rao and J. J. Hwang, Techniques and Standards for Image, Video and Audio coding, Upper Saddle River, N.J.: Prentice Hall, 1996. References continued
[15] M. Ghanbari, Video Coding: an introduction to standard codecs, London, U.K.: Institution of Electrical Engineers, 1999. [16] I.E. G. Richardson, H.264 and MPEG-4 video compression: video coding for next generation multimedia,Chichester: Wiley, 2003. [17]Test streams obtained from ftp://ftp.tek.com/tv/test/streams/Element/MPEG-Video/525/ and [18] Y-J. Chuang, Y-C. Huang and J-L Wu, An efficient block algorithm for splitting an 8x8 DCT into four 4x4 modified DCT used in AVC/H.264, EURASIP 2005, pp [19] P. Assunco and M. Ghanbari, Post Processing of MPEG-2 coded video for transmission at lower bit rates, Proc. IEEEICASSP, pp , Atlanta, GA, 1996. [20] T. Shanableh and M. Ghanbari, Transcoding Architectures for DCT domain heterogeneous video transcoding, Proc. IEEE ICIP, Vol. 1, pp , Thessaloniki, Greece, Sept. 2001,. [21] J. Xin, M.T. Sun and K. Chun, Motion re-estimation for MPEG-2 to MPEG-4 simple profile transcoding, Proc. Int. Workshop Packet Video, Pittsburgh, PA, Apr [22] D-Y. Chan,S-J. Lin and C-Y. Chang, A rate control scheme using Kalman filtering for H.263, Journal of Visual Communication and Image Representation, Vol. 16, pp , Dec [23] S. Liu andA. Bovik, Foveated embedded DCT domain video transcoding, Journal of Visual Communication and Image Representation, Vol. 16, pp , Dec [24] I.E. G. Richardson, Video codec design: developing image and video compression systems,Chichester: Wiley, 2002. [25] G. Sullivan, T. Wiegand and A. Luthra, Draft of Version 4 of H.264/AVC (ITU-T Recommendation H.264 and ISO/IEC (MPEG-4 part 10) Advanced Video Coding), JVT Doc., 14th Meeting: Hong Kong, China Jan [26] G. F-Escribano et.al., Computational complexity reduction of intra frame prediction in MPEG2/H.264video transcoders,ICME, pp ,July 2005. References continued
[27] I. Ahmad et. al., Video transcoding: an overview of various techniques and research issues, IEEE Trans. on multimedia, vol. 7, pp , Oct [28] S. Benyaminovich, O. Hadar and E. Kaminsky, Optimal transrating via DCT coefficients modification and dropping,ITRE, pp , June 2005. [29] J-R. Ohm, Advances in scalable video coding, Proc. IEEE, Vol. 93, pp , Jan [30] J. Wang et. al., An AVS to MPEG-2 transcoding system,Proc. of ISIMP, pp , Oct [31] J. McVeigh et. al., A software based real-time MPEG-2 video encoder, IEEE Trans. CSVT, Vol. 10, pp , Oct