Multiview Rendering

MULTIVIEW RENDERING

Juraj ObertFEL ČVUT

supervised by Ing. Jan Buriánek

2

Outline

Motivation Related Work Rendering Techniques Requirements PerspectiveCube Results Future Work Discussion

Multiview Rendering - Juraj Obert - FEL CVUT

3

Motivation

Perspective (pinhole) camera is defined by a set of rays, which are (1) incident with the center of projection and (2) the optical axis is perpendicular to the image plane.

Relaxation of the latter requirement gives rise to central cameras, which are defined as sets of rays, which meet in one point.


Object

Inverted image

Pinhole

Optical axis

4

Motivation


Perspective (pinhole) camera is defined by a set of rays, which are (1) incident with the center of projection and (2) the optical axis is perpendicular to the image plane.

Relaxation of both requirements gives rise to non-central cameras, which are defined as sets of arbitrary rays in 3D space. In traditional rendering systems, perspective cameras are rather well-explored. Our goal was to create a system for working with non-perspective (also termed multiperspective) projections and images.

5

Related Work – Multiperspective Panoramas

Sweeping a window across the image simulates movement in 3D space

Used in Walt Disney’s Pinocchio in 1940 and researched in 1997 by Wood et al.

Fostered a lot of new research


6

Related Work – X-Slits

X-Slits consist of rays incident with two non-intersecting lines in 3D space Yield images, which are fundamentally different from perspective images,

but look alike (hardly distinguishable for inexperienced users) Can simulate perspective projections by using spatiotemporal volumes


7

Related Work – Digital Cubism


8

Rendering Techniques

Spatiotempor

al volumes

•Stack video frames to create a spatiotemporal volume•Define a projection surface•Shade the projection surface by slicing the spatiotemporal volume•Amenable to programmable graphics hardware•Simple support for animation•Requires input video

Ray-

based co

mposition

•Each pixel corresponds to a different camera ray•Rays are arbitrary, no explicit patterns such as fans of rays as in the case of perspective projections•Implementation on graphics hardware an open research problem•Aliasing problems and user interface problems


9

Spatiotemporal Volume Extract frames from input video file, stack them and build

a volume texture (x, y, t) Three important slicing planes:

x-y original frames y-t pushbroom image x-t epipolar plane image

Can be used to generate various classes of multiperspective images, such as X-Slits, cyclographs, etc.

Our goal was to develop a tool, which would allow a user to do this in real time

Multiview Rendering - Juraj Obert - FEL CVUTx

y

t

10

Requirements

Production requirements Capacity to handle huge amounts of data Utilization of graphics hardware Presets for commonly used camera classes Interactivity Extensibility

Naive approach Decompress video file, create a volume texture containing every single frame

and store it in video memory Example:

19 seconds, 640 x 480, RGB, 24 FPS 400 MB 2 hours, 1280 x 720, RGB, 24 FPS 450 GB !!!

Video memory capacity of latest ( March 2007) NVIDIA graphics cards – 768 MB Video memory capacity of upcoming (June 2007) ATI graphics cards – 1024

MB => naive approach will not work for high-definition content !!!


11

PerspectiveCube - Overview Octree-based spatial organization

of spatiotemporal volumes

GPU memory management based on adaptive assembly of shuffle textures

On-demand loading

Two-level caching mechanism and multithreaded rendering engine for interactivity

Predefined camera types and animation curves

Robust, scalable and GPU-based


12

PerspectiveCube – Rendering


13

PerspectiveCube - Caching


14

Perspective Cube – Demo


15



16



17

Results - Redmond

Pushbroom

X-Slits

Redmond X-Slits right

Original


x

t

Resolution: 360 x 240Length: 148 secondsFPS: 1524 bpp--------------------------------STV size 548 MBShuffle size ~ 128 MB

18

Results – Langweil’s Prague

Original Pushbroom


Resolution: 720 x 480Length: 30 secondsFPS: 15

24 bppSTV size 445 MBShuffle size ~ 64 MB

19

Future Work Explicit camera ray representation

Depart from spatiotemporal volumes and use ray-based techniques More freedom vs. higher productivity

Non-planar projection surfaces Even though already supported in code, practical applications still require

thorough research Problems with unwrapping

Integration with existing modeling systems Access to ray tracing engine, which is necessary for ray-based rendering Unwrapping for non-planar projection surfaces

User interfaces Controls for perspective camera manipulation are too restrictive

for multiperspective cameras How would one map one ray per pixel ?

Performance Effect of used video compression schemes, frame extraction scheduling

algorithm, video memory management Bottleneck is shuffle texture assembly and frame decompression


20

Conclusion

Ray-based techniques amenable to ray tracing engines, but not to programmable graphics devices

PerspectiveCube – system for GPU slicing of spatiotemporal volumes Octree-based spatial organization and smart texture assembly algorithm

for video memory waste minimization Interactivity due to two-level caching mechanism and multithreaded

rendering Scalability in terms of video memory size Support for animation Presets for commonly used multiperspective camera types

In the future Push for ray-based techniques as ray tracing hardware gets developed Don’t necessarily require that programmable graphics devices be used,

CPU can be faster sometimes


21

Thank you

Questions ?


Documents

Multiview Rendering