View
13.214
Download
0
Category
Preview:
DESCRIPTION
Introduction and Description of Error Resiliency and Concealment in ISO/IEC Video Codec H.264 MPEG4 Part10
Citation preview
Error Resiliency & Concealment in H.264
Code from jm9.8/x264
http://getonebyone.com/H264.html
“To enable the compressed bit-stream to better resist channel errors”
OR
“Error resilience techniques enable the compressed bit-stream to resist channel errors so that the
impact on the reconstructed image quality is minimal”
What is Error Resiliency?
But…
some part of the information is inevitably lost and the goal of error concealment, at the receiving side, is to estimate the losses and conceal them in the displayed video.
Error Resiliency Compression
1
Because, generally, the error resiliency schemes introduce some redundancy in the data. On the other hand, compression schemes aim to remove various redundancies from the data
In H.264 we can find several parameters that can be tuned so that a trade-off between compression rate and error resilience can be made targeting different type of problems found in heterogeneous environments.
Tradeoff
H.264 Eco SystemHeterogeneous Environments
Transport
EntropyCoding
DataPartition
NAL
internet
Packets(nal-units or slices)
NALEntropy
DecodingDecode
Prediction Info, Quantized Transform Coeffs….
VLC or AC
VLC CodedData
Coded MBs
Coded Slice/Partition
Packets(nal-units or slices)
Encode
Abstract Level Flow Diagram
Resilience & Concealment in Heterogeneous Environment
Error Propagation A transmission error such as a bit error or packet loss may cause a
video decoder to lose synchronization with the sequence of decoded VLCs.
This can cause the decoder to decode incorrectly some or all of the information after the occurrence of the error and this means that part or all of the decoded VOP will be distorted or completely lost (i.e. the effect of the error spreads spatially through the VOP, ‘spatial error propagation’).
If subsequent VOPs are predicted from the damaged VOP, the distorted area may be used as a prediction reference, leading to temporal error propagation in subsequent VOPs.
Spatial and Temporal Error Propagation
Error Position
Damaged area
Forward Prediction
Error Resilience Employed w.r.t Profiles
Baseline profile includes some enhanced error resilience tools – Flexible Macroblock Ordering (FMO), – Arbitrary Slice Ordering (ASO), and – Redundant Slices (RS)
Extended profile adds further error resilience support in the form of data partitioning (DP).
Main profile does not include enhanced error resilience tools like FMO, ASO, RS, DP, SP or SI Slices.
Error Resiliency Schemes
Flexible Macroblock Ordering (FMO) Arbitrary Slice Ordering (ASO) Data Partitioning (DP) Redundant Slices (RS) SP/SI frame for bitstream switching Reference Frame Selection Intra-block refreshing by R-D control. Random Macroblock Intra Refresh
Flexible Macroblock Ordering (FMO)
Why FMO? Because it aims at avoiding error propagation from a corrupted
packet to subsequent packets.
Objective? The objective behind FMO is to scatter possible errors to the whole
frame as equally as possible, to avoid error accumulation in a limited region.
How it works? It randomizes the data prior to transmission, so that if a segment of
data is lost (e.g. a packet or several packets), the errors are distributed more randomly over the video pictures, rather than causing corruption of a complete regions, making it more likely that relevant neighboring data is available for concealment of lost content.
FMO cont..
Randomizing Data To achieve this, each MB is statically assigned to a slice
group using Macroblock Allocation Map (MBAmap).
How does a MBAmap look like? The MBA_Map is a data structure that maps the spatial
address of a MB to a slice group, in a raster-scan order.
FMO cont..
MBA_Map in H.264 supports 7 arbitrary shapes: 1. Interleaved2. Dispersed3. Foreground & Background4. Box-out5. Raster scan6. Wipe7. Explicit
The parameters that control the mapping functions are included in a Picture Parameter Set (PPS) which is sent to the receiver.
FMOInit
FmoGenerateMapUnitToSliceGroupMap
FmoGenerateMBAmap
FmoGenerateType0MapUnitMap
FmoGenerateType1MapUnitMap
FmoGenerateType2MapUnitMap
FmoGenerateType3MapUnitMap
FmoGenerateType4MapUnitMap
FmoGenerateType5MapUnitMap
FmoGenerateType6MapUnitMap
FMO cont..
Main Encode_one_frame()
Frame_picture()
Field_picture()
FMOInit
Arbitrary Slice OrderingWhy ASO? Since each slice of a coded
picture can be decoded independently of the other slices in the picture, the H.264 design enables sending and receiving the slices of the picture in any order relative to each other.
OR Slices in a coded frame may
follow any decoding order.
sender receiver
Slice # 0
Slice # 1
Slice # 2
0
20
40
Slice Size Criteria Encoder intelligently groups MBs into a slice whose size
is less than (or equal to) the size of MTU (Maximum Transportation Unit).
MTU represents the largest size of a packet that can be transported through networks without being split.
Arbitrary Slice Ordering
Data partitioning enables unequal error protection (UEP) according to syntax elements’ importance, since some syntax elements in the bitstream are more important than others.
The coded data that makes up a slice is placed in three separate Data Partitions (A, B and C), each containing a subset of the coded slice.
Each Partition can be placed in a separate NAL unit and may therefore be transported separately.
Information regarding the Data Partitioning is put into the slice header.
Data Partition (DP)
Data Partition (DP)DP A: Contains header information (MB types, quantization params,
and motion vectors). With the loss of data of DP A, data of the other two partitions
becomes useless.
DP B: Contains Intra CBPs and transform coefficients of I-blocks. The loss of this part will severely impair the recovery of successive
frames due to error propagation, because intra frame and intra-MBs are used as references.
DP C: Contains Inter CBPs and coefficients of P-blocks. Compared to the DPs A and B, the data contained in DP C is less
important. However, it is the biggest partition of a coded slice because a large number of frames are coded as p-frames.
Error concealment mechanism implemented by the decoder
*A = header info *B = Intra CBPs and I-block trans-coeffs*C = Inter CBPs and P-block trans-coeffs
Concealment in Data Partitioning
Error Concealment API Encoder & Decoder Both Have the Same DataPartition Struct
//! DataPartitiontypedef struct datapartition{
Bitstream *bitstream; EncodingEnvironment ee_cabac;
int (*writeSyntaxElement)(SyntaxElement *, struct datapartition *); /*!< virtual function;actual method depends on chosen data partition and entropy coding method */
} DataPartition;
//! Slicetypedef struct{DataPartition *partArr; …..} Slice
Code_a_picturre
Encode_one_Slice()Start_slice()
Generates Slice (Partiton) Header
Main Encode_one_frame()
Frame_picture()
Field_picture()
Data Partition (DP)
SliceGroupId, Picture *, TotalCodedMBs
Picture *
Picture *
Picture *
Picture *
#define MAXSLICEPERPICTURE 100typedef struct { int no_slices; Slice *slices[MAXSLICEPERPICTURE]; ……} Picture;
Redundant Slices
Why? In order to enhance robustness in the event of loss of slices
due to packet drops in network during transmission
Usage In normal operation, the decoder reconstructs the frame
from ‘primary’ pictures and discards any redundant pictures. However, if a primary coded picture is damaged, the decoder may replace the damaged area with decoded data from a redundant picture if available.
A redundant slice is another representation of one or more MBs in the same bitstream.
Encoder put redundant slice information in PPS
SP- /SI- Synchronization/ Switching Frame
SP-/SI mechanism is designed for the purpose of video bitstream switching
It is regarded as an important error resiliency feature in network environments with feedback channel
This scheme assumes that more than one video bitstreams are being sent through network.
In the case of synchronization-lost or presence of error in bitstream, the decoder can signal the presence of error in the frame(s) via feedback channel and the SP frame can be generated for error recovery.
SP/SI-Switching H.264/AVC uses SP- frames that enable random
switching and its info is put in ImageParameters struct
Same valueEn
cod
er
Dec
od
er
Feedback channel
Reference Frame Selection
The reference frame selection mode can be applied both with and without feedback channel (channel which gives lost-packet report)
The reference frame selection with feedback channel uses the reference frame based on the feedback acknowledgement
Reference Frame Selection
There are four possible modes defined in reference frame mode
1. Neither:no back-channel data is returned from the decoder to the encoder.
2. ACK: the decoder returns only acknowledgment messages. In this mode the encoder uses only an acknowledged segment as a reference for inter-frame encoding.
3. NACK: the decoder returns only non-acknowledgment messages. Whenever NACK is received for a frame, the new reference frame is selected from amongst the frames coded before the NACK frame.
4. ACK+NACK: the decoder returns both acknowledgment and non-acknowledgment messages. In the ACK+NACK mode, the encoder switches between two kinds of modes according to the upstream messages from the decoder.
The error propagation and drift due to predictive coding can be eliminated by periodically inserting intra-coded MBs in the
bitstream.
Adaptive Intra Refresh (AIR):H.264 uses intelligent intra-block refreshing by R-D (rate-distortion) control, such that an appropriate block coding option o* will be selected that minimizes the Lagrangian cost function, unlike the periodic or random intra refresh in earlier standards (e.g., MPEG-4, H.263)
Rate Distortion Optimization
Here,o = block coding mode (inter/intra mode and block size)
D(o) = distortion introduced by encoding with mode o and is computed by the SAD (sum of absolute difference) in low complexity mode and by SSD (sum of squared difference) in high complexity mode
R(o) = corresponding coding rate, whereas for inter-block mode it represents the block residual and corresponding motion vector(s).
λ = Lagrange parameter for appropriate weighting of rate and distortion.
Intra Block Refreshing by R-D Control
double RDCost_for_8x8blocks (
int* cnt_nonz, // --> number of nonzero coefficients int64* cbp_blk, // --> cbp blk double lambda, // <-- lagrange multiplier
int block, // <-- 8x8 block numberint mode, // <-- partitioning mode
short pdir, // <-- prediction direction short ref, // <-- reference frame short bwd_ref // <-- abp type)
Error Concealment API
Desc: Calculates Cost function to find the appropriate block coding option. Return: Get Rate-Distortion Cost
Intra Block Refreshing by R-D Control
In order to enhance robustness to data loss, the H.264/AVC design contains a new ability to allow an encoder to send redundant representations of regions of pictures.
void RandomIntraInit(int xsize, int ysize, int refresh)
Desc: Initializes Random Intra module.
xsize, ysize = size of picture in MBs
refresh= refresh rate in MBs per picture
void RandomIntraNewPicture()
Desc: Selects new set of MBs for forced Intra based on the RefreshPattern initialized in RandomIntraInit()
Random Intra MB Refresh
Main Init_img() RandomIntraInit()
int xsize, int ysize, int refresh
Code_a_picturreMain Encode_one_frame()
Frame_picture()
Field_picture()
Picture *
Picture *
Picture *
Picture *
RandomIntraNewPicture()
Called, only after initialization (or changes) of the picture size or the random intra refresh value, immediately after parsing config file
Allocates forced INTRA MBs
Random Intra MB Refresh
Error Concealment
Decoder Error Concealment
Decode ALL Frames(main)
Docode_One_Frame
( Flag, DecParams, DecOutputs )
Read_New_Slice
( DecParams, DecInputs, DecOutputs )
Flag is the state variable for error detector and concealor
( DecParams, DecInputs, DecOutputs )
Recommended