5LSH0 Advanced Topics Video & Analysis - TU/evca.ele.tue.nl/sites/default/files/5LSH0 mod 02 3D intro multiview video comp.pdf · – 3D Coding architecture concept – Depth signals

1

1

PdW-SZ-EB / 2016

Fac. EE SPS-VCA

Adv. Topics MMedia Video / 5LSH0 /

Mod 02 3D intro & multiview video

Advanced Topics Multimedia Video

(5LSH0), Module 02

3D Geometry, 3D Multiview Video Coding & Rendering

Peter H.N. de With, Sveta Zinger & Y. Morvan( [email protected] )

2

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Multiview 3D video / Outline

∗ Camera geometry

– Intrinsic/extrinsic camera parameters

– Camera calibration

∗ 3D Video coding & Multiview rendering

– 3D Coding architecture concept

– Depth signals and depth estimation

– 3D Multiview video coding

– 3D rendering: algorithm and artifacts removal

3

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



5LSH0 Advanced Topics Video &

Analysis

A. Projective geometry - Introduction

Sveta Zinger

Video Coding and Architectures Research group, TU/e( [email protected] )

4

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



∗ Projective geometry

– branch of geometry dealing with the properties and

invariants of geometric figures under projection

– serves as a mathematical framework for 3D multi-view

imaging, 3D computer graphics

• image formation process modeling

• image synthesis

• reconstruction of 3D objects from multiple images

Introduction – (1)

5

PdW-SZ-EB / 2016

Fac. EE SPS-VCA




Two parallel rails intersect in the image plane at the vanishing point

6

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



∗ Euclidian geometry

– Usually used to model lines, planes or points in 3D

∗ Why do we need projective geometry?

– easier to model intersection of parallel lines at infinity

– perspective scaling operation requires division in

Euclidian geometry => non-linearity => better to avoid


2

7

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Homogeneous coordinates – (1)

Define a point

in Euclidian space In projective space

3-element vector 4-element vector

(X, Y, Z)T (X1, X2, X3,, X4)T

Inhomogeneous coordinates Homogeneous coordinates

8

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Homogeneous coordinates – (2)

∗ Inhomogeneous coordinates (X, Y, Z)T and

homogeneous coordinates (X1, X2, X3,, X4)T are related

∗ Mapping from n-dimensional Euclidian space

0,,,434241

≠=== 4XwhereXXZXXYXXX

( ) ( ) ,,,...,,,...,,2121 4444 34444 2144 344 21

spaceprojective

T

n

spaceEuclidian

T

nXXXXXX λλλλ→

where – free scaling parameter, or homogeneous scaling

parameter

0≠λ

9

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Pinhole camera model – (1)

1. Coordinate frame aligned with camera center.

2. Image in focal plane (between object and camera).

3. Allows projection of 3D object onto 2D image.

positiveimage

10

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



3D from multiple images: concept

Cloud of points is projected

into multiple images from

different viewpoints.

Can we reverse the projection

process and reconstruct the

points?

11

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



3D from multiple images: some

images from input sequence

http://www.cs.unc.edu/~marc/tutorial.pdf

12

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



3D from multiple

images: algorithm

3

13

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



3D from multiple images:

result

PhD thesis of Ping Li, TUe VCA

http://vca.ele.tue.nl/people/PLi_publ.html

14

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Mod 03 Multiview geometry & coding

B. Camera Geometry

15

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Camera geometry / Concept – (1)∗ To understand the 3D structure of

objects/scene, a relation between

point coordinates in the 3D world

to pixels position is required.

∗ We have 3 coordinates systems:

image, camera and world.

∗ Goal: map a point in 3D space (the

world coordinate system) to the

image plane (the image

coordinates system).

16

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Camera geometry / Concept – (2)

∗ Link world, camera and image coordinates by a set of

parameters known as intrinsic and extrinsic parameters.

∗ Intrinsic parameters:

• focal length,

• width and height of the pixel on the sensor,

• position of the principal point (origin of the image coordinates

system).

∗ Extrinsic parameters:

• camera position,

• camera orientation.

17

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Camera geometry / Image formation

∗ Remember: pinhole camera model

Projection of point onto the image plane

results in point

18

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Central projection using homogeneous coordinates

With homogeneous coordinates, the central projection can be written as a

linear equation.

The central projection maps the 3D space to 2D space.

4

19

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Camera geometry / CCD camera

∗ Require more general conversion to pixel

coordinates

– Principal point offset in pixel units.

– Conversion from camera coordinates to pixel

coordinates is obtained with width/height,

hence fraction of the pixels.

20

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Extrinsic camera parameters

∗ Extrinsic parameters define orientation and location of

the camera in the world coordinate system.

∗ Involves Euclidean transform between world and camera coordinates

- R is a 3x3 rotation matrix

- t is a 3D translation vector

21

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Projective camera / Summary – (1)

∗ We can finally map the 3D point to the image

22

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Projective camera / Summary – (2)

∗ Combining the camera matrix (intrinsic parameters) and

the rotation/translation matrix (extrinsic parameters) we

obtain the camera calibration matrix .

∗ Projection matrix 3x4 has 11 degrees of freedom (scaling

invariance).

intrinsic

parameters

extrinsic

parameterscamera calibration

matrix

world

coordinatespixel

coordinates

23

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Camera calibration

∗ Goal: estimating coefficients of the

camera calibration matrix.

– Once the camera calibration matrix

parameters are known, the camera is

“calibrated”.

∗ Simple calibration algorithm

– It is assumed that the world coordinates of

points are known with their corresponding pixel

coordinates.

– Points are usually arranged in a special pattern

for easy calibration.

24

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Linear method for estimating matrix C – (1)

∗ World-point coordinates and image-pixel positions are

linked with the camera calibration matrix

∗ is the 3x4 projection matrix, can be written as

∗ Algorithm consists of two steps:

– 1. Compute matrix C with a set of know 3D positions and their

respective positions in the image.

– 2. Extrinsic and intrinsic parameters are estimated from C.

5

25

PdW-SZ-EB / 2016

Fac. EE SPS-VCA




∗ Use world point coordinates and their corresponding

pixel coordinates in the image to determine .

– Each correspondence generates two equations

which can be written as

26

PdW-SZ-EB / 2016

Fac. EE SPS-VCA




∗ Stack equations into one equation system:

∗ The equation has 12 unknown parameters: at least 6

correspondence points are required.

∗ Typically, more points are used.

– Equation system gets over-constrained.

– Equation is then solved using a least squares minimization.

27

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Application: back-projection of points to rays

∗ Given a point x in an image, we determine the set of

points in 3D space that map to this point.

∗ This ray is presented as the joint of two points:

– the camera center where

– the point where is the pseudo-inverse of

∗ The ray is then formed by the joint of these two points:

28

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Mod 02 Multiview 3D Geometry & Coding

C. Multiview 3D TV coding architecture

29

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



� Presented work was initiated to support the development of

video compression algorithms for 3D video systems.

� We present a 3D video system architecture based on new

approaches for

− Depth estimation : Acquisition of 3D content

− New coding techniques for the efficient storage and

transmission and,

− Rendering of 3D video.

Introduction multiview coding30

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



� The MPEG community has a considerable interest in

standardizing technologies for 3D and FTV applications, e.g.

− 3D TV enabling the perception of depth using a multi-view display,

− free-viewpoint video that allows the viewer to interactively select a

viewpoint of the scene.

MVC intro / Applications of 3D video

6

31

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



� Several 3D video representation formats are explored:

− 1-texture+1-depth format,

− N-texture video format and,

− N-texture+N-depth that was adopted in our work.

3D video Representation Formats32

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Multi-view video

acquisitiondecoding and rendering

depthestimation

H.264 multi-view

H.264 multi-viewtexture Coder

H.264 multi-viewdepth Decoder

H.264 multi-viewtexture Decoder

3Drendering

-view

synthesis

Multi-view video

compression

netw

ork

A

B

CD

∗ The proposed 3D video system architecture is composed of:

– a depth-estimation sub-system (3D acquisition),

– an H.264 multi-view video coder

– an H.264 multi-view depth video coder

– a 3D-video rendering engine

3D Video Coding Architecture

33

PdW-SZ-EB / 2016

Fac. EE SPS-VCA




D. 3D Depth Estimation

and 3D Rendering

34

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



table of matching-costs

admitted depth

table of matching-coststable of matching-coststable of matching-costs

Depth estimation using 2 views – (1)∗ A popular method: calculate depth for each scanline using a 1D

optimization

∗ Algorithm summary:

– for each scanline, calculate a table of matching costs.

– Optimize matching-cost table using dynamic programming (ref. Viterbi).

35

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Depth estimation using 2 views – (2)36

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Depth estimation using multiple views

To estimate accurate depth images, we propose two

new constraints.

∗ Instead of estimating depth images pair-wise, we propose

to employ all views simultaneously.

∗ To avoid scanline artifacts, we employ an inter-scanline

cost that enforces smooth variations of depth (smoothness

constraint).

7

37

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Depth estimation using multiple views38

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



� A pixel position in the left view can be

predicted from its corresponding position

in the right view using the image

warping equation

� Disadvantage: generates holes in the

rendered image, i.e. occluded pixels.

� Requires calibration parameters

Rendering of multi-view images – (1)

39

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



� Relief texture mapping expresses the image warping equation

Rendering of multi-view images – (2)

∗ Advantages of relief texture are that the:

– Pre-warping step performs horizontal and vertical pixel-shift

combined with pixel re-sampling, thus resolving occluded

pixels,

– Post-warping equation corresponds to a planar texture mapping

operation, thus efficiently implemented in a GPU.

40

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



View Rendering Example - Original

41

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



View Rendering Example – Rendered 42

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



3D from multiple images:

video demonstration

http://www.cs.unc.edu/~marc/

This video shows:

• input image sequence

• reconstructed camera

positions, cloud of points

• obtained depth map

• texture mapped on

depth

8

43

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



View rendering: Free-ViewPoint (FVP)44

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Challenges of FVP:

cracks due to image sampling

3D Warping

Reference image Virtual image

Possible solution: median filtering

45

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Challenges of FVP:

poorly defined borders => contour artifacts

Possible solution: label edges and delete them after warping

46

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Challenges of FVP:

disocclusions inpainting

Possible solution: fill in the disoccluded pixels

with background texture information

47

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



View rendering: free-viewpoint result

PhD thesis research

Luat Do, TUe VCA

http://vca.ele.tue.nl/people/LDo_publ.html

48

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



References

– Y. Morvan, “Acquisition, Compression and Rendering of

Depth and Texture for Multi-view Video”, Ph.D. thesis,

Eindhoven University of Technology, 2009

– http://mathworld.wolfram.com/ProjectiveGeometry.html

9

49

PdW-SZ-EB / 2016

Fac. EE SPS-VCA




E. 3D Coding of multiview images

50

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Predictive coding of multiview images

∗ For an efficient transmission, an independent compression of correlated

camera-views should be avoided.

∗ One predictive-coding algorithm investigated is based on an image

rendering technique.

∗ The idea followed is to render an image as seen by the predicted

camera using Depth Image Based Rendering (DIBR).

51

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



� Reminder: the adopted video format is N-texture+N-depth.

� Coding of multi-view depth images: render depth image at the position of a

predicted camera.

Predictive coding of depth images52

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Synthetic view

central

camera-view

Including view rendering in H.264 coding

∗ For the compression of multiple views, we have integrated the

view-rendering algorithm into an H.264 encoder.

∗ A camera view (texture or depth) is predicted using either

» the central reference camera, or

» the synthetic rendered view.

53

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Coding structure simulcast Coding structure with random access

NB: There is a trade-off between coding efficiency and random-access!

Coding structure for random access

∗ The coding structure defines which view can be employed as reference

(predictor) for compression.

∗ For free viewpoint video, the coding structure should allow random

access to an arbitrary view.

54

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Texture is temporally stable:

ME slightly outperforms

view-synthesis prediction at

the loss of random access.

Coding Results of ‘Breakdancers’ seq. – (1)

The proposed multi-view

coding system provides:

- random access to

arbitrary views

- coding performances

similar to Simulcast

coding.

10

55

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



Depth is not temporally

stable: ME does not

work

Coding results of ‘Breakdancers’ seq. – (2)56

PdW-SZ-EB / 2016

Fac. EE SPS-VCA



∗ The presented 3D video processing and coding system

– performs accurate depth estimation by employing

simultaneously multiple views,

– renders high-quality images by appropriately handling

occluded pixels

– achieves efficient compression by exploiting inter-view

redundancy for the texture and depth images.

∗ The coding system relies on H.264 and thus allows a gradual

introduction of cost-efficient 3D systems.

Multiview Coding / Conclusions

Documents

5LSH0 Advanced Topics Video & Analysis - TU/evca.ele.tue.nl/sites/default/files/5LSH0 mod 02 3D intro multiview video comp.pdf · – 3D Coding architecture concept – Depth signals