Creating Adaptive Views for Group Video Teleconferencing – An Image-Based Approach Creating Adaptive Views for Group Video Teleconferencing – An Image-Based

Creating Adaptive Views for Group Creating Adaptive Views for Group Video Teleconferencing Video Teleconferencing

– An Image-Based Approach – An Image-Based Approach

Ruigang YangRuigang Yang Celso Kurashima Celso Kurashima Andrew Nashel Andrew Nashel Herman Towles Herman Towles Anselmo Lastra Anselmo Lastra Henry FuchsHenry Fuchs

Ruigang YangRuigang Yang Celso Kurashima Celso Kurashima Andrew Nashel Andrew Nashel Herman Towles Herman Towles Anselmo Lastra Anselmo Lastra Henry FuchsHenry Fuchs

University of North Carolina University of North Carolina

at Chapel Hillat Chapel Hill

University of North Carolina University of North Carolina

at Chapel Hillat Chapel Hill

International Workshop on Immersive Telepresence 2002 Slide 2

Current Teleconferencing

Capture

TransportDisplay

?


The Office of the Future


Group Teleconferencing

• Multiple persons Multiple persons (3-4) at each site(3-4) at each site

• Life-size, Life-size, monoscopic monoscopic displaydisplay

• High-resolution High-resolution seamless seamless imageryimagery

• Active view Active view controlcontrol

• Multiple persons Multiple persons (3-4) at each site(3-4) at each site

• Life-size, Life-size, monoscopic monoscopic displaydisplay

• High-resolution High-resolution seamless seamless imageryimagery

• Active view Active view controlcontrol

Cameras


Active View Control

Provide the best approximating view


Active View Control

A view synthesis problemA view synthesis problem• Extract 3D geometry from a Extract 3D geometry from a

few camerasfew cameras– Less expensiveLess expensive– Hard to get good resultsHard to get good results

• Image-based method: Image-based method: capture many imagescapture many images– Looks really good on every Looks really good on every

scenescene– Need many imagesNeed many images

A view synthesis problemA view synthesis problem• Extract 3D geometry from a Extract 3D geometry from a

few camerasfew cameras– Less expensiveLess expensive– Hard to get good resultsHard to get good results

• Image-based method: Image-based method: capture many imagescapture many images– Looks really good on every Looks really good on every

scenescene– Need many imagesNeed many images


Our Image-based Approach

• Observation:Observation:Eye level remains relatively the same Eye level remains relatively the same

during a conference sessionduring a conference session

• A compact Light Field representationA compact Light Field representation– Parameterized by a 3D function (s, u, v)Parameterized by a 3D function (s, u, v)

• Observation:Observation:Eye level remains relatively the same Eye level remains relatively the same

during a conference sessionduring a conference session

• A compact Light Field representationA compact Light Field representation– Parameterized by a 3D function (s, u, v)Parameterized by a 3D function (s, u, v)

u

v

s

t

Focal Plane


Linear Light Field


LLF Rendering

• Projective Texture mapping and blendingProjective Texture mapping and blending– Tessellate the focal planeTessellate the focal plane

– Project input images onto the focal planeProject input images onto the focal plane

– View-dependent blendingView-dependent blending

• Projective Texture mapping and blendingProjective Texture mapping and blending– Tessellate the focal planeTessellate the focal plane

– Project input images onto the focal planeProject input images onto the focal plane

– View-dependent blendingView-dependent blending

New view

Base image

Focal Plane


Blending Function

V

C0 Ci CN-1C1

0

1

D

i

N-1

j j

ii

ii

ˆ

ˆ

expˆ2

2Focal Plane2


Samples Images

PerspectiveProjection

OrthogonalProjection

(extreme case)


Sampling Analysis

• Configuration parameters– Focal plane depth D

– Camera’s FOV

– Camera’s horizontal resolution W

– Inter-camera distance d

• Error term: pixel drift (e)

• Configuration parameters– Focal plane depth D

– Camera’s FOV

– Camera’s horizontal resolution W

– Inter-camera distance d

• Error term: pixel drift (e)

Given the configuration parameters, and a desired error tolerance e , what is the maximum depth deviation D from the optimal depth D.

Given the configuration parameters, and a desired error tolerance e , what is the maximum depth deviation D from the optimal depth D.


Sampling Analysis – Result


More results

• Distributed SystemDistributed System– 11 cameras11 cameras

– 6 capture PC ( 640x480)6 capture PC ( 640x480)• ROI encodedROI encoded• JPEG compressionJPEG compression

– One rendering PCOne rendering PC• Roughly 1000 x 480 Roughly 1000 x 480

outputoutput• 4-7 frames per second4-7 frames per second

• Distributed SystemDistributed System– 11 cameras11 cameras

– 6 capture PC ( 640x480)6 capture PC ( 640x480)• ROI encodedROI encoded• JPEG compressionJPEG compression

– One rendering PCOne rendering PC• Roughly 1000 x 480 Roughly 1000 x 480

outputoutput• 4-7 frames per second4-7 frames per second


Conclusions

We presented a novel system designed We presented a novel system designed specifically for specifically for groupgroup video- video-teleconferencing.teleconferencing.

• Best approximate view for the groupBest approximate view for the group

• Photo-realistic results at interactive rate Photo-realistic results at interactive rate

• Flexible and scaleableFlexible and scaleable

We presented a novel system designed We presented a novel system designed specifically for specifically for groupgroup video- video-teleconferencing.teleconferencing.

• Best approximate view for the groupBest approximate view for the group

• Photo-realistic results at interactive rate Photo-realistic results at interactive rate

• Flexible and scaleableFlexible and scaleable


Acknowledgements

Funding support from The Department of Energy's ASCI VIEWS Program

Sandia National Laboratories USA

Collaborators from SandiaCollaborators from SandiaPhil Heermann

Christine Yang

Corbin Stewart

Funding support from The Department of Energy's ASCI VIEWS Program

Sandia National Laboratories USA

Collaborators from SandiaCollaborators from SandiaPhil Heermann

Christine Yang

Corbin Stewart


The End

Thank You

Documents

Creating Adaptive Views for Group Video Teleconferencing – An Image-Based Approach Creating Adaptive Views for Group Video Teleconferencing – An Image-Based