Upload
buitruc
View
232
Download
0
Embed Size (px)
Citation preview
September 25, 2013 Sam Siewert
CS A490 Digital Media and Interactive
Systems
Lecture 5 - Deeper Dive into MPEG Digital Video Encoding
Overview MPEG at a High Level
– Pixel/Color Encoding in YUV (e.g. YCrCb 4:2:2) – Macro block encoding (DCT, Quantization, Zigzag, Huffman) – Group of Pictures
Deeper Dive in 13818-1 and 13818-2 – 13818-1: Transport Streams for Video & Audio
Container for Program Streams (188 Byte Packets) Multiplexed Video and Audio Elementary Streams PSI – Program Specific Information System Clock (PCT, PTS/DTS)
– 13818-2: Elementary Video Stream Encode/Decode Video DCT Macro-blocks Color Format GoP (I-Frame, B-Frame, P-Frame) Motion Compensation and Vector Quantization
Differences Between MPEG-2 and MPEG-4 Trick-Play Operations on Program and Transport Streams
Sam Siewert 2
MPEG-2: Order Of Operators
Sam Siewert 3
#1: POINT (Pixel) Encoding #2 A-C: Macro-Block Lossy Intra-Frame Compression #3: Motion-Based Compression in Group of Pictures
#1
#2A
#2B #2C #3
Sam Siewert 4
Step #1 – RGB to YCrCb 4:4:4 24-bit/30-bit (Lossless)
For every Y sample in a scan-line, there is also one CrCb sample – Each Y (Y7:Y0), Cr (Cr7:Cr0), & Cb (Cb7:Cb0) Sample is 8 bits – No compression between RGB and YCrCb 4:4:4 (both 24 bits/pixel) – For DCI and Post Production, Each Sample can be 10 bits
Typically a Post Production, CEDIA or DCI format
… 0 319
… 76,480 76,799
…
= Y, Cr, and Cb sample = Y sample only
48 bit to 32 bit
Sam Siewert 5
Step #1 – RGB to YCrCb 4:2:2 (Lossy) For every 2 Y samples in a scan-line, there is one CrCb sample – Each Y (Y7:Y0), Cr (Cr7:Cr0), & Cb (Cb7:Cb0) Sample is 8 bits – Two RGB Pixes = 48 bits, Whereas Two YCrCb is 32 bits, or 16
bits per pixel vs. 24 bits per pixel (33% smaller frame size)
… 0 319
… 76,480 76,799
…
= Y, Cr, and Cb sample = Y sample only
Pixel-0 = Y7:Y00, Cb7:Cb00; Pixel-1 = Y7:Y01, Cr7:Cr00 Pixel-2 = Y7:Y02, Cb7:Cb01; Pixel-3 = Y7:Y03, Cr7:Cr01 Pixel-4 = Y7:Y04, Cb7:Cb02; Pixel-5 = Y7:Y05, Cr7:Cr02
Sam Siewert 6
Step #1 – RGB to YCrCb 4:2:0 (Lossy) For every 4 Y samples in a scan-line, there is one CrCb sample – Each Y (Y7:Y0), Cr (Cr7:Cr0), & Cb (Cb7:Cb0) Sample is 8 bits – Two RGB Pixes = 48 bits, Whereas Four YCrCb is 48 bits, or 12
bits per pixel on average vs. 24 bits per pixel (50% smaller)
… 0 319
… 76,480 76,799
…
= Cr, Cb sample = Y sample only
Convolution Concepts Math operation on 2 functions, that produces a 3rd PSF Sharpen meets this Definition So do Many Mask Operations applied to Pixel Neighborhoods
Sam Siewert 7
2 impulses, f(t), g(X – t)
Area inside intersection
f convolved with g over t
DCT – Discrete Cosine Transform Convolution of Image with Discrete Cosine See http://www.cse.uaa.alaska.edu/~ssiewert/a490dmis_code/example-dct1/ De-convolved to restore image from Convolved Image
Sam Siewert 8
DCT
Inverse DCT
DCT Concepts F(x) is a sum of sinusoids (with specific frequency and amplitude) DCT operates of a discrete number of samples We can derive DC sum at any x, even where F(x) not known N x N Macro-block has Zero Frequency DC at 0,0 Increasing Horizontal Frequency Increasing Vertical Frequency Can De-convolve Can Eliminate High Frequency Horizontal and Vertical Terms with Minimal Losses to De-convolution (Loss of High Frequency Image Features)
Sam Siewert 9
Basic Concept of Waveforms Complex Waveform is Sum of Simple Fundamentals Simple Fundamentals Can Be Derived from Complex
Sam Siewert 10
Scanline DCT Example Small Losses Due to DCT, iDCT Numerical Truncation Larger Losses Due to H.O.T. Quantization and Truncation http://www.cse.uaa.alaska.edu/~ssiewert/a490dmis_doc/1D-DCT-N-Fundamentals.xlsx
Sam Siewert 11
What Is Lost with DCT Quantization? Noise More Than Anything Else Complex XY Variable Patterns (Real Science Data?)
Sam Siewert 12
Complex Tiling Higher Frequency X Higher Frequency Y Terms Can Still be Ignored
Complex Wood Texture Most Detail in X Far Less in Y
Randomized Cactus Image High X Detail High Y Detail Most Loss of Detail, But Noisy
Step #2A: Macro-block Discrete Cosine Transform
8x8 Pixel Block – Macro-block – SD NTSC 720x480 (90x60 Macro-blocks), 3:2 Aspect Ratio – HD 720 1280x720 (160x90 Macro-blocks), 16:9 AR – HD 1080 1920x1080 (240x135 Macro-blocks), 16:9 AR
Sam Siewert 13
Step #2B: Macro-block Quantization (Lossy)
Apply Weighting and Scaling 8x8 to DCT Produces Lots of Repeated Values (and Zeros) Compared to Original
Sam Siewert 14
Decode Process for #2A-B
Sam Siewert 15
How Lossy is the Decode Macro-Block?
Sam Siewert 16
OpenCV Macroblock DCT Example Same Cactus 320x240 with 80x80 DCT Macroblocks
Sam Siewert 17
DCT iDCT
Same Cactus 320x240 Again with 8x8 DCT Macroblocks
DCT iDCT
Mathematics for 2D DCT Frequency Variation on X and Y axes from top left to bottom right Straight-forward Algorithm Based on 2D Equation is O(n2) per dimension Like Cooley-Tukey for DFT, a DCT Algorithm that is O(n*log2(n)) has been formulated (Arai, Y.; Agui, T.; Nakajima, M. - Numerical Recipes: The Art of Scientific Computing (3rd ed.)) http://www.cse.uaa.alaska.edu/~ssiewert/a490dmis_code/dct2/dct2.c
Sam Siewert 18
http://en.wikipedia.org/wiki/File:Dctjpeg.png
Step #2C: Macro-block Run-Length and Huffman Encoding
Zig-Zag Run-Length Encoding to Exploit Repeated Data and Zeros found in H.O.T. of Quantized DCT
– 86, 1, 7, -5, -1, 0, 1, 0, 0, 2, -1, 1, 0, -1, 0 , 0, 0, 0, -1, 0, 0, …
Becomes:
Sam Siewert 19
Huffman Applied to RLE Data Huffman Tables for MPEG-2 Macro-Blocks Defined in 13818-2 (Lossless) Compression Based on Probability of Occurance Shannon’s Source Coding Theory: log2(P), P=probability of occurrence, Binary encoding of Symbols
Sam Siewert 20
Step #3: Group of Pictures Concept – Transmit Change-Only Data I-Frame Compressed Only Intra-Frame By Methods #2A-2C to Macro-Blocks I-Frame Can Be Decoded Alone P-Frame is Differences Only Over the GoP B-Frame is Differences Only Between Both I-Frame and Closest P-Frame Difference Data Can be Further Encoded with Lossless Methods Without Steps 2A-C, Specifically Quantization, and With High Motion Video, Could Blow-Up
Sam Siewert 21
Group of Pictures: High Level View
Sam Siewert 22
Overall MPEG YCrCb Compression Performance
Standard Definition 720x480x2 (675KB/frame) @ 30fps – Requires 20MB/sec (200 Mbps) Uncompressed – Typical MPEG-2 @ 3.75 Mbps, > 50x Compression – Typical MPEG-4 @ 1.5 Mbps, > 100x Compression – 10 to 20 Programs on QAM 256 (48Mbps, 6MhZ/Ch) – ≈10 MPEG-4 Programs on ATSC 8VSB (19.39 Mbps, 6MhZ/Ch)
HD 720p (1280x720x2,1800KB/frame) @ 30fps – Requires 53MB/sec (530Mbps) Uncompressed – Typical MPEG-2 @ 20 Mbps, > 25x Compression – Typical MPEG-4 @ 10 Mbps, > 50x Compression
HD 1080p (1920x1080x2, 4050KB/frame) @ 30fps – Requires 120MB/sec (1200Mbps) Uncompressed – Typical MPEG-2, VC-1 @ 45 Mbps, > 30x Compression – Typical MPEG-4 @ 20 Mbps, > 60x Compression
Sam Siewert 23
13818-2 Defines Elementray/Program Streams
13818-2: Elementary Video Stream Encode/Decode – Defines Color Sub-Sampling Formats – 8x8 Macro-Block Encoding – Video DCT – Post DCT Macro-Block Quantization Weighting and Scaling
Coefficients – RLE Zig-Zag Macro-Block Sampling – Huffman Encoding Table – Group of Pictures:I-Frame, B-Frame, P-Frame – Presentation and Decode Time Stamps (PTS/DTS) – Order of Encode and Decode Operations – Format for Video and Audio Elementary Streams
Not Suitable for Transport over Lossy Networks, but Sufficient for Local Playback (DVD, PC HDD, Flash-Memory Media)
Sam Siewert 24
13818-1 13818-1: Transport Streams for Video & Audio – Container for Program Streams (188 Byte Packets) – Multiplexed Video and Audio Elementary Streams – PSI – Program Specific Information
PID – Program ID Guide Data Emergency Broadcast
– System Clock (PCR, PTS/DTS) – Sequence Headers – Resolution and Format Information, Bit-Rate
GoP Header, Frame Header Slices of Macro-Blocks for Resoltuion Decoder Information (Color, Quantization Tables)
– Can Be Multiple Programs or Combined Audio and Video as a Program
MPEG-2 Video Elementary Stream AC-3 Audio Elementary Stream Secondary Audio Stream (Different Langauge) Up to 10 or More Audio+Video in One Transport Stream for Virtual Channles
Sam Siewert 25
Parsing an Elementary Video Stream
Sam Siewert 26
Many 188-Byte Packet Types and Header Allows for Multi-plexing of many Video and Audio Streams on a Carrier
PSI (Program Specific Information) Data
There are 4 PSI tables: Program Association (PAT), Program Map (PMT), Conditional Access (CAT), and Network Information (NIT). The MPEG-2 specification does not specify the format of the CAT and NIT.
PAT stands for Program Association Table. It lists all programs available in the transport stream. Each
of the listed programs is identified by a 16-bit value called program_number. Each of the programs listed in PAT has an associated value of PID for its Program Map Table (PMT). The value 0x0000 of program_number is reserved to specify the PID where to look for Network Information Table (NIT). If such a program is not present in PAT the default PID value (0x0010) shall be used for NIT. TS Packets containing PAT information always have PID 0x0000.
Program Map Tables (PMTs) contain information about programs. For each program, there is one
PMT. While the MPEG-2 standard permits more than one PMT section to be transmitted on a single PID, most MPEG-2 "users" such as ATSC and SCTE require each PMT to be transmitted on a separate PID that is not used for any other packets. The PMTs provide information on each program present in the transport stream, including the program_number, and list the elementary streams that comprise the described MPEG-2 program. There are also locations for optional descriptors that describe the entire MPEG-2 program, as well as an optional descriptor for each elementary stream. Each elementary stream is labeled with a stream_type value.
Sam Siewert 27
PSI (Program Specific Information) Data
PCR To enable a decoder to present synchronized content, such as audio tracks matching the
associated video, at least once each 100 ms a Program Clock Reference, or PCR is transmitted in the adaptation field of an MPEG-2 transport stream packet. The PID with the PCR for an MPEG-2 program is identified by the pcr_pid value in the associated Program Map Table. The value of the PCR, when properly used, is employed to generate a system_timing_clock in the decoder. The STC decoder, when properly implemented, provides a highly accurate time base that is used to synchronize audio and video elementary streams. Timing in MPEG2 references this clock, for example the presentation time stamp (PTS) is intended to be relative to the PCR. The first 33 bits are based on a 90 kHz clock. The last 9 are based on a 27 MHz clock. The maximum jitter permitted for the PCR is +/- 500 ns.
Null packets Some transmission schemes, such as those in ATSC and DVB, impose strict constant
bitrate requirements on the transport stream. In order to ensure that the stream maintains a constant bitrate, a Multiplexer may need to insert some additional packets. The PID 0x1FFF is reserved for this purpose. The payload of null packets may not contain any data at all, and the receiver is expected to ignore its contents.
Sam Siewert 28
MPEG-4 vs. MPEG-2 MPEG-2 – Defined by ISO 13818-1, 13818-2 – Leverages MPEG-1 (Motion Picture Experts Group – 1988) – Widely Used for Digital Video – Digital Cable TV, DVD – Transport Stream designed for Broadcast (Lossy, No Beginning or End of
Stream) ATSC – Advanced Television Systems Committee (HDTV Broadcast)
– 8VSB Modulation – 8 level Vestigal Sideband Modulation, 6MhZ channel, 19.39 Mbps, Reed-Solomon Error Correction
– Up to 1080p (1920x1080) Video Resolution – AC-3 (Dolby) Audio
DVB – Digital Video Broadcast (Europe, Satellite) – Program Stream designed for Playback Media (DVD, Flash, HDD, etc.)
MPEG-4 – Defined by ISO 14496 (1998) – Leverages MPEG-2 Standards for Program/Transport, Encode/Decode – Better Compression Rates (improved motion prediction for P,B frames),
MPEG-4 Part-10 (H.264), e.g. Blu-Ray – Extensions for Digital Rights Management – Advanced Audio Encoding – Becoming More Widely Deployed for HD and Because of Lower Bit-Rate
Transport Streams
Sam Siewert 29
Trick-Play Program Stream Operations – Pause – Single Frame Display, Silent – Fast-Forward – 60fps (2x), 120fps (4x), etc., Silent – Rewind – 2x, 4x, etc. Frame Rate in Reverse, Silent – Start-Over – Back to First Frame – Play – 30fps (29.97 NTSC), Audio
Program Stream – Video and Audio Elementary Streams – PTS/DTS – Presentation and Decode Time Stamps – Adjust as
Needed – Bit-rate Control? (Doubles with 2x FF?)
Transport Stream – Multiple Programs Containing Multiple Elementary Streams – PCR – Program Clock Reference – Adjust As Needed – Discontinuity Bit – Set As Needed
Sam Siewert 30