M16758 – Influence of views and depth compression onto quality

of synthesized views

July, 2009, London

Krzysztof WegnerKrzysztof Klimaszewski

Marek Domański

team supervisor: Marek DomańskiChair of Multimedia Telecommunications and Microelectronics

Poznań University of Technology, Poland

M16758 – Influence of views 2

Multiview system

RectificationCorrection

Depth M apEstim ation

V iewSynthesis

Rendering

V iew and Depth M ap

Transm ission

D isplay

Com

pres

sion

Dec

ompr

essi

on

M16758 – Influence of views 3

Multiview system

RectificationCorrection

Depth M apEstim ation

V iewSynthesis

Rendering

V iew and Depth M ap

Transm ission

D isplay

Com

pres

sion

Dec

ompr

essi

on

M16758 – Influence of views 4

Depth map compression issues

How to compress depth maps? Can MVC be used?

How to share bandwidth between views and depth maps? Which one is more important? What is the optimal ratio?

M16758 – Influence of views 5

Depth map features Differences between depth maps and

viewsDepth Maps Viewsonly one color

componentthree color

componentslimited texture abundance of texture

prominent contoursless prominent

contoursaccurate depth value

importantaccurate value not

necessary

M16758 – Influence of views 6

Depth map decimation What is the influence of depth map

decimation on the view synthesis process?

1024

76 8

x2

1024

76 8

512

384

256

192x4

M16758 – Influence of views 7

Test setup

M VCC om pression

ViewSynthesis

A

B

C

M VCC om pression

M VCD ecom pression

M VCD ecom pression

D epth M aps

ViewsView A

View C

Depth M ap A

Depth M ap C

Transm

ission B'

M16758 – Influence of views 8

View quality measurement

M VCC om pression

ViewSynthesis

A

B

C

M VCC om pression

M VCD ecom pression

M VCD ecom pression

D epth M aps

ViewsView A

View C

Depth M ap A

Depth M ap C

Transm

ission B'

M16758 – Influence of views 9

Cameras and software The following software and camera

numbers were used:

Data setOriginal pair (A

and C)Synthesized

view (B)Book

Arrival7 and 10 8

Pantomime 38 and 41 39Newspaper 3 and 6 4

VSRS version 3DERS version 3JMVC version 3

M16758 – Influence of views 10

Results Synthesis quality for UNCOMPRESSED

view and depth map

Data setPSNR using

uncompressed data [dB]

PSNR using uncompressed data

(depth maps decimated with factor 2)

[dB]

Book Arrival 36.1787 36.1865

Pantomime 35.5908 35.6526

Newspaper 29.2191 29.3317

M16758 – Influence of views 11

Results Results for compressed data – Book

Arrival

0 500000 1000000 1500000 2000000 2500000 3000000 3500000

31

32

33

34

35

36

37

PSNR of synthesized view against total bitrate

2121 dec 2x21 dec 4x2626 dec 2x26 dec 4x3131 dec 2x31 dec 4x3636 dec 2x36 dec 4x

Bitrate [bps]

PS

NR

[dB

]

M16758 – Influence of views 12

Results Results for compressed data – Book

Arrival

-0.1 0.0999999999999999 0.3 0.5 0.7

31

32

33

34

35

36

37

PSNR of synthesized view against bitrate ratio

2121 dec 2x21 dec 4x2626 dec 2x26 dec 4x3131 dec 2x31 dec 4x3636 dec 2x36 dec 4x

Ratio

PS

NR

[dB

]

M16758 – Influence of views 13

Results Results for compressed data –

Pantomime

1000000 2000000 3000000 4000000 5000000 6000000 7000000 8000000 9000000 10000000 11000000

33

33.5

34

34.5

35

35.5

PSNR of synthesized view against total bitrate

2121 dec 2x2626 dec 2x3131 dec 2x3636 dec 2x

Bitrate [bps]

PS

NR

[dB

]

M16758 – Influence of views 14

Results Results for compressed data –

Pantomime

0 0.1 0.2 0.3 0.4 0.5 0.6

33

33.5

34

34.5

35

35.5

PSNR of synthesized view against bitrate ratio

2121 dec 2x2626 dec 2x3131 dec 2x3636 dec 2x

Ratio

PS

NR

[dB

]

M16758 – Influence of views 15

Results Results for compressed data –

Newspaper

0 1000000 2000000 3000000 4000000 5000000 6000000 7000000 8000000

27.7000

27.9000

28.1000

28.3000

28.5000

28.7000

28.9000

29.1000

29.3000

PSNR of synthesized view against total bitrate

2121 dec 2x2626 dec 2x3131 dec 2x3636 dec 2x

Bitrate [bps]

PS

NR

[dB

]

M16758 – Influence of views 16

Results Results for compressed data –

Newspaper

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

27.7000

27.9000

28.1000

28.3000

28.5000

28.7000

28.9000

29.1000

29.3000

PSNR of synthesized view against bitrate ratio

2121 dec 2x2626 dec 2x3131 dec 2x3636 dec 2x

Ratio

PS

NR

[dB

]

Subjective tests

A single-stimulus test was conducted 15 spectators quality was rated from

5 (“imperceptible artifacts”) to 0 (“very bad with annoying artifacts”)

The original video from actual real camera was used as hidden reference.

M16758 – Influence of views 17

Subjective quality 1/3

M16758 – Influence of views 18

Subjective quality 2/3

M16758 – Influence of views 19

Subjective quality 3/3

M16758 – Influence of views 20

Optimal QD for depth map

Depth quantization parameter index QD has negligible (less than 0.2 dB decrease) impact on synthesized view quality as long its value is smaller than the following threshold estimated from the experiments

M16758 – Influence of views 21

T hresho ld QD 49 0.005 76 QP 2

M16758 – Influence of views 22

Conclusions 1/2 it is possible to transmit the depth maps using

MVC codec with sufficient quality view quality is much more important than depth

quality in terms of synthesized view quality depth bitrate can be kept as low as 10 - 20 % of

total bitrate without significant loss of synthesized view quality (for decimated depth map)

M16758 – Influence of views 23

Conclusions 2/2

for a certain range of values of quantization parameter index for depth compression (QD) , no change of synthesized view quality is noticeable

higher compression rate with comparable quality can be obtained by decimating the depth map by a factor of 2

M16758 – Influence of views 24

Recommendations We recommend to consider usage of

decimated depth map in future depth map coding experiments.