View
0
Download
0
Category
Preview:
Citation preview
Perceptual Quality Driven 3-D Video over Networks
C.T.E.R. Hewage
Supervisors: Dr. Stewart Worrall and Dr. Safak Dogan
2
Outline Introduction
Aim and Objectives
Efficient Coding Approaches for Stereoscopic Video
Objective Quality Measures for Stereoscopic Video
Efficient Transmission Strategies for Backward Compatible Stereoscopic Video Applications
Error Concealment Techniques for 3-D Video
Conclusion
Future Works
List of Publications
3
Introduction
3-D Video – Provides the sense of “being there” 3-D Video communications – Provides more natural
conditions for human interaction Tele-presence, Tele-immersion, 3D TV
NodeB
RNC
LTE
WiMAX
3DTV
Mobile 3DTV
4
Stereoscopic Video
The left and right views are fused in the visual cortex of the brain to perceive the depth of a scene
•Advantages of stereoscopic video over other representations of 3-D video
• Simple representation (easy camera arrangement)
• Cost effective display systems
• Easy to adapt for existing audio-visual communication technologies
5
Stereoscopic Video Capture
Left and Right View Camera Simple
Cost effective
Colour plus Depth Camera
128
255
0
Znear
Zfar(a) (b)
(a)Colour image
(b)Per-pixel depth image.
6
Problem Definition
3-D Video delivery beyond conventional video requires enormous system resources such as storage and bandwidth.
Development of 3-D video technologies start from the beginning would not be a feasible and cost effective solution.
The initial development of 3-D video technologies should be backward compatible for existing 2-D video applications.
Quality evaluation of 3-D video can be done accurately using subjective evaluation tests.
The transmission aspects of 3-D video has not received much attention to date.
7
Aim and Objectives Aim: To enable backward compatible 3-D
video services over bandwidth limited and unreliable communication networks using a perceptual quality driven approach.
Objectives: Explore efficient methods of encoding 3-D video using existing
method of 2-D video compression
Propose objective quality metrics for coded 3-D video in a range of compression ratios and packet loss rates.
Propose, develop and test methods of error resilience over errorprone wireless channel conditions.
Propose, develop and test error concealment methods for 3-D video data transmitted over unreliable communication networks
8
Outline Introduction
Aim and Objectives
Efficient Coding Approaches for Stereoscopic Video
Objective Quality Measures for Stereoscopic Video
Efficient Transmission Strategies for Backward Compatible Stereoscopic Video Applications
Error Concealment Techniques for 3-D Video
Conclusion
Future Works
List of Publications
9
Efficient Encoding Approaches for Stereoscopic Video
Exploration of effcient encoding configurations for 3-D video using existing video coding standards Compression efficiency
Compatibility with end-to-end communication technologies
Asymmetric coding of colour plus depth video Coarsely quantized depth images
Temporarly down-sampled depth images
Performance analysis of stereoscopic video transmission over IP
10
Exploration of efficient encoding configurations
Parallel encoding configurations
Single-output encoding configurations
H.264/AVC
CODEC
H.264/AVC
CODECColour
Depth
Colour
Base Layer
EnhancementLayerDepth
MultiLayer
CODEC
Side by Side Image
Colour plus Depth
H.264/AVC
Encoder
11
Stereoscopic video encoding with MPEG4-MAC
Global MAC Bit-
Stream
MPEG-4
MAC
EncoderColour
Depth
Shape
•A single output bit-stream•A monochrome depth map•The binary shape of the video object need to be transmitted•Backward compatibility
H.264/SVC
EncoderBase Layer
EnhancementLayer
Colour
Depth
Global SVC
Bit-stream
•A single output bit-stream•Backward compatibility•Asymmetric coding support•Possibility of layered depth images (LDI)•Inter layer prediction can be used depending on the correlation of the stereo image pair
12
“Orbi” Colour (Left) and Depth (Right) video sequences
R-D Performance of SSV coding using MPEG-4 MAC, H.264/AVC and scalable H.264/AVC
13
Efficient Encoding Approaches for Stereoscopic Video
Exploration of effcient encoding configurations for 3-D video using existing video coding standards Compression efficiency
Compatibility with end-to-end communication technologies
Asymmetric coding of colour plus depth video Coarsely quantized depth images
Temporarly down-sampled depth images
Performance analysis of stereoscopic video transmission over IP
14
Asymmetric coding of colour plus depth stereoscopic video
Mixed-resolution encoding concept for stereoscopic video is based on the response of Human Visual System (HVS).
Asymmetric coding of colour and depth
map video??? Coarsely quantized depth maps
Temporally down-sampled depth maps
15
Coarsely Quantized Depth Map Sequences
514843383328231813Mode 5
514641363126211611Mode 4
49443934292419149Mode 3
47423732272217127Mode 2
45403530252015105Mode 1
Depth
45403530252015105Colour
QPType of video
For depth image coding at the enhancement layer, five Quantization Parameter (QP) modes are used. A similar set of QP values are used to obtain R-D results for the colour image coding at the base layer.
Coarse Grain Scalability (CGS) of scalable H.264/AVC is utilized to obtain different R-D performances for colour and depth sequences
•JSVM Version 8.13
•IPPP…IPP… sequence
•I frame every 75 frames
16
Image Quality and Percentage Depth Bitrate @ overall bitrate of 1Mbps
38.4436.24Depth using Mode 5
39.5337.15Depth using Mode 440.6438.08Depth using Mode 3
41.6238.77Depth using Mode 2 42.6439.25Depth using Mode 1 37.4837.01Colour
InterviewOrbiY-PSNR (dB)
Image Sequence
0.1436.8736.81Mode 50.1836.9436.87Mode 40.2436.9536.90Mode 30.3136.7436.70Mode 20.4036.3236.30Mode 1
RightLeft
Percentage depth bitrate
(%)
Image Quality: PSNR (dB)Mode
0.1737.1136.98Mode 50.2237.1937.14Mode 40.2837.0736.99Mode 30.3436.8736.86Mode 20.4136.7636.70Mode 1
RightLeft
Percentage depth bitrate
(%)
Image Quality: PSNR (dB)Mode
17
Efficient Encoding Approaches for Stereoscopic Video
Exploration of effcient encoding configurations for 3-D video using existing video coding standards Compression efficiency
Compatibility with end-to-end communication technologies
Asymmetric coding of colour plus depth video Coarsely quantized depth images
Temporarly down-sampled depth images
Performance analysis of stereoscopic video transmission over IP
18
Stereoscopic Video Performance Over IP
The error sensitivities of different components of immersive media may be diverse in nature.
The losses in colour image sequence may have greater impact on the perceived quality than the damage caused by a corrupted depth image sequence.
The error performance analysis can be utilized to propose novel UPA and UEP mechanisms for 3-D video communication applications.
19
Stereoscopic Video Performance Over IP
Orbi Interview
20
Outline Introduction
Aim and Objectives
Efficient Coding Approaches for Stereoscopic Video
Objective Quality Measures for Stereoscopic Video
Efficient Transmission Strategies for Backward Compatible Stereoscopic Video Applications
Error Concealment Techniques for 3-D Video
Conclusion
Future Works
List of Publications
21
Objective Quality Measures for Stereoscopic Video
3-D video quality = {image quality, depth perception, presence, naturalness, eye strain, etc.}
Subjective evaluation tests are widely used to obtain human response for 3-D perception. Time consuming.
Enormous effort and large number of subjects are required.
Controlled test environments are required.
Does objective quality measures of 3-D video represent the true quality as perceived by human observers?
Objective quality metrics for 3-D video enable researchers, developers to evaluate the quality more efficiently.
22
Objective Quality Measures for Stereoscopic Video
The 3-D video quality is measured using subjective and objective quality metrics for a range of compression rates and packet loss rates. Subjective tests are conducted according to the ITU-
Recommendation BT.1438 (using DSCQS method).• Two perceptual attributes namely, overall image quality and depth
perception are measured subjectively. PSNR, SSIM and VQM measures of colour, depth and rendered left
and right image sequences are considered as objective quality measures.
Then the relationships (correlations) between objective and subjective quality ratings are derived using a symmetrical logistic function as described in the following.
p = 1 / [1 + exp ( D - DM ) G ]
where, p is the normalized opinion score, D is the distortion parameter, DM and G are constants and G may be positive or negative.
23
Quality of Asymmetrically Coded Colour Plus Depth Map Video
Orbi Breakdance
24
Correlation Between Objective and Subjective Quality Measures
25
Correlation Between Objective and Subjective Quality Measures
0.21730.06590.71950.55500.10540.7528Average SSIM of the Rendered Left and Right Views
0.76020.12330.01880.79100.21170.0015Depth SSIM0.21120.06500.72740.53470.10340.7618Colour SSIM
0.12150.04930.84320.27740.07450.8764Average VQM of the Rendered Left and Right
0.12240.04900.84210.24600.07010.8904Colour VQM
0.17290.05880.77680.59360.10900.7356Average PSNR of the Rendered Left and Right views
0.77500.21190.000010.77140.12420.0043Depth PSNR0.14010.05290.81920.42650.09240.8100Colour PSNR
SSERMSECCSSERMSECC
Depth PerceptionOverall Image QualityObjective Quality Model
26
Perceived Quality under Transmission Errors
Orbi Breakdance
27
Correlation Between Objective and Subjective Quality Measures
0.142500.08900.63100.28270.12530.5099Average SSIM of Rendered Left and Right Views
0.14740.09050.61830.29270.12750.4926Colour SSIM
0.065540.06030.83030.11280.07920.8045Average VQM of Rendered Left and Right
0.080390.06680.79180.12410.08300.7848Colour VQM
0.043100.04890.88840.11230.07890.8054Average PSNR of Rendered Left and Right views
0.044260.04960.88540.11580.08020.7992Colour PSNR
SSERMSECCSSERMSECC
Depth PerceptionOverall Image QualityObjective Quality Model
28
Outline Introduction
Aim and Objectives
Efficient Coding Approaches for Stereoscopic Video
Objective Quality Measures for Stereoscopic Video
Efficient Transmission Strategies for Backward Compatible Stereoscopic Video Applications
Error Concealment Techniques for 3-D Video
Conclusion
Future Works
List of Publications
29
Efficient Transmission Strategies for Backward Compatible Stereoscopic Video Applications
3-D video should be protected over unreliable and error
prone communication channels
UEP is an effective method to provide unbalanced
protection for video delivery over networks.
However adding redundant information to 3-D video
stream is not a feasible approach.
Hence in this study, UEP is implemented by
varying the transmission power for colour and depth map streams
mapping colour and depth map bit-streams into different medium
access priority classes.
30
Efficient Transmission Strategies for Backward Compatible Stereoscopic Video Applications
In this study colour bit-stream is assigned higher protection compared to the depth bit-stream
Why?
31
Unequal Error Protection for Backward Compatible 3-D video Transmission over WiMAX
The UPA module dynamically allocates differentiated transmissionpower for colour and depth map streams based on the generated lookup tables.
The colour bit-stream will be survived under transmission errors than the depth map stream due to the high power associated with the colour video packets.
3-D video input
WiMAX
TransmissionSystem
Decoder
Depth video Packets
Colour video Packets3-D Video
Encoder based on
SVC
Extractor Module
Power control
UPAModule
Feedback
32
Performance of the UEP Scheme
8 8.5 9 9.5 10 10.5 11 11.5 12 12.5 13 13.517
19
21
23
25
27
29
31
33
35
37
39
40.5
SNR (dB)
Y−
PS
NR
(dB
)
Colour without UEPColour with UEPDepth without UEPDepth with UEP
8 9 10 11 12 1322
24
26
28
30
32
34
36
38
40
42
SNR (dB)Y
−P
SN
R (
dB)
Colour without UEPColour with UEPDepth without UEPDepth with UEP
Orbi Interview
33
Performance of the UEP Scheme
Without UEP With UEP
Without UEP With UEP
34
Performance of the UEP Scheme
Perception of Overall Image Quality Using SSCQS Method
0.5
1
1.5
2
2.5
3
3.5
9.95 10.5 11.05 11.6 12.15 12.7 13.25
SNR (dB)
MO
S
Overall Image Quality with UEP Overall Image Quality without UEP
35
Performance of the UEP Scheme
Depth Perception using SSCQS Method
0.5
1
1.5
2
2.5
3
3.5
9.95 10.5 11.05 11.6 12.15 12.7 13.25
SNR (dB)
MO
S
Depth Perception with UEP Depth Perception without UEP
36
Outline Introduction
Aim and Objectives
Efficient Coding Approaches for Stereoscopic Video
Objective Quality Measures for Stereoscopic Video
Efficient Transmission Strategies for Backward Compatible Stereoscopic Video Applications
Error Concealment Techniques for 3-D Video
Conclusion
Future Works
List of Publications
37
Error Concealment Techniques for 3-D Video
3-D error concealment methods can utilize the information from the same video stream as well as from the corresponding video stream to recover the missing information. A Novel Frame Concealment Method for Depth Maps
Using Corresponding Colour Motion VectorsCompression efficiency
Error Concealment Scheme for Stereoscopic Video Using the Shared Motion Information Send by the Encoder
3-D Video Concealment Using Associated Shape Information
38
Frame Concealment
128
255
0
Znear
Zfar(a) (b)
0 10 20 30 40 50 60 70 80 90 100−0.05
0
0.05
0.1
0.15
0.2
Frame Number
r(A
,B)
Interview
Horizontal componentVertical component
39
Depth Frame Concealment
MVs
. . .
. . .
Depth frame n Depth frame n+1
Colour frame n Colour frame n+1
Proposed method
JSVM-FC
40
GlobalSVC Bitstream
Error Concealment Scheme for Stereoscopic Video Using the Shared Motion Information
SVC Encoder
Colour MVs
Base layer Encoder
Enhancement layer encoder
Colour
Depth
. . .
. . .
MVsMVs
#1 Depth frame #2 Depth frame
#1 Colour frame #2 Colour frame
40.2741.0537.2539.69
Proposed MV
sharing scheme
42.3441.0538.1040.13Separate MVs
Depth (dB)
Colour (dB)
Depth (dB)
Colour (dB)
Interview-PSNROrbi-PSNR
Method
41
Performance of the Proposed MethodOrbi
Colour Depth
42
Performance of the Proposed Method
Orbi Sequence @ 5% PLR with individual MVs
Orbi Sequence @ 5% PLR with shared MVs
43
Outline Introduction
Aim and Objectives
Efficient Coding Approaches for Stereoscopic Video
Objective Quality Measures for Stereoscopic Video
Efficient Transmission Strategies for Backward Compatible Stereoscopic Video Applications
Error Concealment Techniques for 3-D Video
Conclusions
Future Works
List of Publications
44
Conclusions 3-D Video Coding
Compression of Colour and depth using scalable H.264/AVC Asymmetric coding of colour plus depth video Stereoscopic video performance over IP
Objective Quality Metrics for 3-D video Quality Error Resilience Schemes for 3-D Video
UEP scheme for 3-D video transmission over WiMAX networks Prioritization scheme for 3-D video distribution over WLAN
Error Concealment Schemes for 3-D Video Depth frame concealment using associated colour motion
information Frame concealment scheme for 3-D video using shared motion
information 3-D error concealment scheme based on associated shape
information
45
Future Works Rate Adaptive 3D Video Coding
Asymmetric coding approached can be extended to implement open/close rate adaptive schemes for 3-D video transmission.
Generic 3-D Video Quality Metric Extend this quality evaluation to cover the end-to-end
chain of 3-D video technologies from 3-D capture to display.
Associate more perceptual attributes such as presence, naturalness and eye-strain.
The Proposed Error Resilience and Concealment Schemes Can be Extended for Multi-View Video
46
List of Publications1. C.T.E.R. Hewage, S. Nasir, S. Worrall, S. Dogan and A.M. Kondoz, "Prioritized 3-D Video Transmission over IEEE
802.11e", submitted to 2008 IEEE International Symposium on Circuits and Systems, Taipei, Taiwan, May 2009.
2. C.T.E.R. Hewage, Z. Ahmad, S. Worrall, S. Dogan and A.M. Kondoz, "Unequal Error Protection for Backward Compatible 3-D Video Transmission over WiMAX", submitted to 2008 IEEE International Symposium on Circuits and Systems, Taipei, Taiwan, May 2009.
3. C.T.E.R. Hewage, S. Worrall, S. Dogan and A.M. Kondoz, "Prediction of stereoscopic video quality using objective quality models of 2-D video", Electronics Letters -- 31 July 2008 -- Volume 44, Issue 16, p. 963-965.
4. C.T.E.R. Hewage, S. Worrall, S. Dogan and A.M. Kondoz, "Quality Evaluation of Colour plus Depth Map Based Stereoscopic Video", submitted to IEEE Journal of Selected Topics in Signal Processing:, May 2008.
5. C.T.E.R. Hewage, S. Worrall, S. Dogan and A.M. Kondoz, "Frame Concealment Algorithm for Stereoscopic Video Using Motion Vector Sharing", Proceedings of IEEE International Conference on Multimedia & Expo 2008 (ICME2008), Hannover, Germany, June 2008, p. 485-488.
6. C.T.E.R. Hewage, S. Worrall, S. Dogan and A.M. Kondoz, "A Novel Frame Concealment Method for Depth Maps Using Corresponding Colour Motion Vectors", Proceedings of 2nd 3DTV conference (3DTV-CON'08-IEEE), Istanbul, Turkey, May 2008, p.149-152.
7. C.T.E.R. Hewage, S. Worrall, S. Dogan, H. Kodikara Arachchi and A.M. Kondoz, "Stereoscopic TV over IP", Proceedings of the 4th IET European Conference on Visual Media Production (CVMP'2007), London, UK, November 2007.
8. C.T.E.R. Hewage, H.A. Karim, S. Worrall, S. Dogan and A.M. Kondoz, "Comparison of Stereo Video Coding Support in MPEG-4 MAC, H.264/AVC and H.264/SVC", Proceedings of the 4th IET International Conference on Visual Information Engineering (VIE'2007), London, UK, 25-27 July 2007.
9. C.T.E.R. Hewage, H. Kodikara Arachchi, T. Masterton, A.C. Yu, H. Uzuner, S. Dogan and A.M. Kondoz, "Content Adaptation for Virtual Office Environment Using Scalable Video Coding", Proceedings of the 16th IST Mobile and Wireless Communications Summit (IST Summit'2007), Budapest, Hungary, 1-5 July 2007.
47
Thank You!..
Any Questions?
Contacts: Chaminda Hewage
E.Thushara@surrey.ac.uk
Recommended