Fast Inter Mode Decision Based on Textural Segmentation and Correlations for Multiview Video Coding

Wei Zhu, Xiang Tian, Fan Zhou and Yaowu Chen

IEEE TCE, 2010

2

Outline

Introduction Observation and Analysis Proposed Fast Inter Mode Decision Experimental Results Conclusion

3

Introduction

Joint Video Team(JVT) has developed a Joint Multiview Video Model(JMVM) which is based on H.264/AVC to compress multiple video data efficiently.

Fig.1 shows an illustration of prediction structure employed in JMVM(Hierarchical B prediction structure).

Primary view Auxiliary view

ME DE

ME +

4

Introduction

MVC uses Rate-Distortion Optimization(RDO) technique to select the optimal coding mode which is time consuming.

The coding structure in MVC are more complex than H.264/AVC, especially for auxiliary views employing inter-view prediction.

The prediction directions could be decided to reduce the computational complexity of auxiliary views.

The correlations between views can be exploited to further speed up the mode decision of MVC.

5

Observation and Analysis

Use JMVM 4.0 with the full mode decision to analyze modes features.

Analysis

Skip mode occupies the largest proportion among all MB modes

a large proportion background no inter estimation, faster determine to be Skip mode as fast as possible

Inter8x8 mode occupies the smallest proportion among Inter modes

consume 50% encoding time reduce the estimation of Inter8x8

6

Observation and Analysis

Inter_DE mode only occupy a small proportion, but the computation time consumed by disparity estimation for inter-view prediction occupies more than 50% of encoding time.

reduce disparity estimation as much as possible

Intra mode occupies a small proportion among all MB modes, and computation is also small.

implement it before Inter mode decision, the results can

be used for the optimization of Inter mode decisions

7

Observation and Analysis

Target: exploit the correlation between Intra mode and Inter mode, and the correlation between textural complexities and RD costs.

Analysis

Both Intra mode and Inter mode have higher RD costs in higher texture region

Both Intra mode and Inter mode have lower RD costs in lower texture region

Inter/Intra mode RD costs are sensitive to textural regions

Intra mode RD costs can be used for the segmentation of textural region

※ Blocks with higher values of RD costs are darker and lower values are lighter.

Intra Inter

8

Proposed Fast Inter Mode Decision

① Reference views’ Intra/Skip RD costs info are transferred to the current view.

② Current frame is segmented into three textural regions.

③ Do Intra, Skip, and Inter16x16 modes.

Analysis

Intra : get its RD cost to obtain the textural region type of MB

Skip : get its RD cost or the early decision of Skip mode

Inter16x16 : get results of ME and DE, and use these results to decide if to do DE/Inter8x8 or not

9

Proposed Fast Inter Mode DecisionA. Segmentation of Textural Regions Reference views’ Intra mode RD costs are used to segment

textural regions. High textural area Intra4x4 mode Low textural area Intra16x16 mode

Intra mode is classified into INTRA4 and INTRA16 INTRA4 : Intra4x4, Intra8x8 and PCM modes high texture INTRA16 : Intra16x16 low texture

10

Proposed Fast Inter Mode Decision Sort RD costs of INTRA4 and INTRA16 from low to high

INTRA16 mode RD cost are in a relative

low level

INTRA4 mode RD cost are in a relative

high level

The variation of RD costs with the same Intra mode among views are consistent

The distribution of all Intra mode RD costs can be divided into three segments the low, the high, and the in-between values

11

Proposed Fast Inter Mode Decision Calculate thresholds of the textural segmentation:

Use neighboring views Intra RD costs

○ k : index of neighboring reference views

Obtain segmentation threshold

For each MB in current frame, obtain its textural region type

○ n : index of MB

12

Proposed Fast Inter Mode Decision Distribution of textural regions in Race1 sequence is shown

in Fig.6

High textural region

Median textural region

Low textural region

Distributions of textural regions are varying with different QP, it is more suitable for Inter mode decision

13

Proposed Fast Inter Mode Decision

Idea : Considering that RD costs have different sensitivities to textural complexities, so the RD cost threshold should be calculated for different textural regions.

Numbers of Skip mode selected by full mode decision:

The numbers are similar between frames with the same QP

B. Early decision of Skip Mode

14

Proposed Fast Inter Mode Decision

15

Proposed Fast Inter Mode Decision Average Skip mode RD cost curves:

low texture

always above high texture

Values are similar at same time instance

16

Proposed Fast Inter Mode Decision Skip mode RD cost includes only distortion part. It is more relevant

to textural complexity of MB. The relationship between and :

Early decision of Skip mode:

① if Skip mode has minimum RD cost among Intra, Skip, Inter16x16.

② Its RD cost is less than its corresponding threshold ()according to textural region type.

Not used for anchor frames

17

Proposed Fast Inter Mode DecisionC. Selection of Disparity Estimation The correlation of selected inter-view prediction between Inter16x16

mode and all Inter modes was analyzed:

Prediction results of INTER16x16 mode can be utilized to guide the selection of DE for other Inter modes

※ and

Inter16x16 All Inter mode

Choose Inter-view prediction Choose temporal prediction

Enabling of disparity estimation for other Inter modes:

1.01.11.2

18

Proposed Fast Inter Mode Decision

A distribution of Inter8x8 mode selected by the full mode decision:

The proportions of Inter8x8 mode in different textural regions:

Most of Inter8x8 mode appear in regions with more textural complexity

There are few Inter8x8 mode in backgrounds with no motion activity

D. Reduction of Inter8x8 Mode Estimation

Most of Inter8x8 mode are in the high textural region, very few in low textural region

19

Proposed Fast Inter Mode Decision Enabling of Inter8x8 mode estimation:

※ is the motion vector of MB

20

Experiment Results

Software version JMVM4.0

Search mode Fast search

Search range 96

QP 22, 27, 32, 37

Test sequence Exit, Ballroom, Race1, Flamenco2, Rena

Number of view 2 or 3

Compare with Full mode decision in JMVC, Shen’s algorithm [1]

[1] L. Q. Shen, Z. Liu, S. Liu, Z. Y. Zhang, and P. An, “Selective disparity estimation and variable size motion estimation based on motion homogeneity for multi-view coding,” IEEE Trans. Broadcasting, vol. 55, no. 4, pp. 761-766, Dec. 2009.

21

Experiment Results For the early decision of Skip mode, parameter was determined by

experimental analysis:

※

22

Experiment Results

Compare to the full mode decision:

23

Experiment Results

Compare to Shen’s algorithm

24

Experiment Results

Compare to Shen’s algorithm

25

Conclusion

A fast Inter mode decision is proposed to reduce the computational complexity of MVC.

Skip mode is early decided, the disparity estimation is selected, and the Inter8×8 mode estimation is reduced.

As compared to the full mode decision about 55~70% time saving with negligible loss of coding efficiency.

The proposed algorithm could be combined with fast motion search and disparity search algorithms to further save the computation time.