1

Transport and Rendering Challenges Transport and Rendering Challenges of Multi-Stream, 3D Tele-of Multi-Stream, 3D Tele-

Immersion DataImmersion Data

Herman Towles,Herman Towles,Sang-Uok Kum, Travis Sparks, Sudipta Sinha, Scott Sang-Uok Kum, Travis Sparks, Sudipta Sinha, Scott

Larsen and Nathan BeddesLarsen and Nathan Beddes

Department of Computer ScienceDepartment of Computer ScienceUniversity of North Carolina at Chapel HillUniversity of North Carolina at Chapel Hill

October 28, 2003October 28, 2003

Transport and Rendering Challenges Transport and Rendering Challenges of Multi-Stream, 3D Tele-of Multi-Stream, 3D Tele-

Immersion DataImmersion Data

Herman Towles,Herman Towles,Sang-Uok Kum, Travis Sparks, Sudipta Sinha, Scott Sang-Uok Kum, Travis Sparks, Sudipta Sinha, Scott

Larsen and Nathan BeddesLarsen and Nathan Beddes

Department of Computer ScienceDepartment of Computer ScienceUniversity of North Carolina at Chapel HillUniversity of North Carolina at Chapel Hill

October 28, 2003October 28, 2003

2

‘’‘’Portal’ to Another Place Portal’ to Another Place

UNC - January 2000

Static, High-Resolution 3D Scene

Head-Tracked, Passive Stereo Projector Display

3

Static Scene Acquisition Static Scene Acquisition

• DeltaSphere™ DeltaSphere™ Scanning Laser Scanning Laser RangefinderRangefinder– Time of FlightTime of Flight

– Resolution: ~3K x 2K for 150Resolution: ~3K x 2K for 150oo azimuth and 90azimuth and 90oo elevation elevation

– Digital Camera mounted at Digital Camera mounted at identical COP for color imageidentical COP for color image

• OutputOutput– Range ImageRange Image

– Color Camera ImageColor Camera Image

• DeltaSphere™ DeltaSphere™ Scanning Laser Scanning Laser RangefinderRangefinder– Time of FlightTime of Flight

– Resolution: ~3K x 2K for 150Resolution: ~3K x 2K for 150oo azimuth and 90azimuth and 90oo elevation elevation

– Digital Camera mounted at Digital Camera mounted at identical COP for color imageidentical COP for color image

• OutputOutput– Range ImageRange Image

– Color Camera ImageColor Camera Image

Images courtesy of 3rd Tech, Inc.

4

NTII CollaboratorsNTII Collaborators

• UNC-CH – Computer GraphicsUNC-CH – Computer Graphics– Henry Fuchs et al.Henry Fuchs et al.

• UPenn – Computer VisionUPenn – Computer Vision– Kostas Daniilidas et al.Kostas Daniilidas et al.

• Brown University – Collaborative Brown University – Collaborative GraphicsGraphics

– Andy van Dam et al.Andy van Dam et al.

• Advanced Network & ServicesAdvanced Network & Services– Jaron Lanier & Amela SadagicJaron Lanier & Amela Sadagic

Goal: Develop a Live 3D Goal: Develop a Live 3D ‘Portal’‘Portal’

• UNC-CH – Computer GraphicsUNC-CH – Computer Graphics– Henry Fuchs et al.Henry Fuchs et al.

• UPenn – Computer VisionUPenn – Computer Vision– Kostas Daniilidas et al.Kostas Daniilidas et al.

• Brown University – Collaborative Brown University – Collaborative GraphicsGraphics

– Andy van Dam et al.Andy van Dam et al.

• Advanced Network & ServicesAdvanced Network & Services– Jaron Lanier & Amela SadagicJaron Lanier & Amela Sadagic

Goal: Develop a Live 3D Goal: Develop a Live 3D ‘Portal’‘Portal’

5

3D Tele-Immersion – Phase 3D Tele-Immersion – Phase 1 1

• 3D Real-Time 3D Real-Time Reconstruction Reconstruction

– Correlation-based Stereo AlgorithmCorrelation-based Stereo Algorithm

– 1-3 fps at 320 x 240 (quad-PC/550)1-3 fps at 320 x 240 (quad-PC/550)

– 1 cubic meter working volume 1 cubic meter working volume

– Up to 5 streams of depth imagesUp to 5 streams of depth images

• 3D Tracked Stereo Display3D Tracked Stereo Display– Life-size with gaze awarenessLife-size with gaze awareness

– 3-PC Rendering Cluster3-PC Rendering Cluster

– Composite SceneComposite Scene

» Live 3D Remote CollaboratorLive 3D Remote Collaborator

» Static, office backgroundStatic, office background

» Collaborative GraphicsCollaborative Graphics

• 3D Real-Time 3D Real-Time Reconstruction Reconstruction

– Correlation-based Stereo AlgorithmCorrelation-based Stereo Algorithm

– 1-3 fps at 320 x 240 (quad-PC/550)1-3 fps at 320 x 240 (quad-PC/550)

– 1 cubic meter working volume 1 cubic meter working volume

– Up to 5 streams of depth imagesUp to 5 streams of depth images

• 3D Tracked Stereo Display3D Tracked Stereo Display– Life-size with gaze awarenessLife-size with gaze awareness

– 3-PC Rendering Cluster3-PC Rendering Cluster

– Composite SceneComposite Scene

» Live 3D Remote CollaboratorLive 3D Remote Collaborator

» Static, office backgroundStatic, office background

» Collaborative GraphicsCollaborative Graphics

6

Standard Videoconferencing Standard Videoconferencing ModelModel

Camera MonitorNetwork

• 1-Camera to 1-Monitor Paradigm 1-Camera to 1-Monitor Paradigm

• Single Stream of 2D Image DataSingle Stream of 2D Image Data

View-Dependent Data !View-Dependent Data !

• 1-Camera to 1-Monitor Paradigm 1-Camera to 1-Monitor Paradigm

• Single Stream of 2D Image DataSingle Stream of 2D Image Data

View-Dependent Data !View-Dependent Data !

7

3D Tele-Conferencing 3D Tele-Conferencing ModelModel

3D Camera 3D RenderingNetwork

• N-Camera to 1-Monitor N-Camera to 1-Monitor

• Multiple Streams of 3D Image DataMultiple Streams of 3D Image Data– Depth Image (16-bit 1/z) and RGB Image (24-bit)Depth Image (16-bit 1/z) and RGB Image (24-bit)

View-Independent Data w/ View-View-Independent Data w/ View-Dependent Rendering !Dependent Rendering !

• N-Camera to 1-Monitor N-Camera to 1-Monitor

• Multiple Streams of 3D Image DataMultiple Streams of 3D Image Data– Depth Image (16-bit 1/z) and RGB Image (24-bit)Depth Image (16-bit 1/z) and RGB Image (24-bit)

View-Independent Data w/ View-View-Independent Data w/ View-Dependent Rendering !Dependent Rendering !

3D Camera

3D Camera

User Tracker

8

3D Tele-Immersion- Phase 23D Tele-Immersion- Phase 2

• Higher QualityHigher Quality– Improve Resolution, Frame Improve Resolution, Frame

Rate and DisplayRate and Display

• Larger Larger Reconstruction Reconstruction VolumeVolume– 30-60 color cameras 30-60 color cameras

– Acquire full environmentAcquire full environment

• Higher QualityHigher Quality– Improve Resolution, Frame Improve Resolution, Frame

Rate and DisplayRate and Display

• Larger Larger Reconstruction Reconstruction VolumeVolume– 30-60 color cameras 30-60 color cameras

– Acquire full environmentAcquire full environment Terascale Computing System (Lemieux)at Pittsburgh Supercomputing Center (PSC)

Acquisition Reconstruction

3D Camera

3D Camera 3D RenderingNetworkReconstructionAcquisition Network

9

2000-2002 Results 2000-2002 Results

• Show videoShow video• Show videoShow video

10

Outline Outline

• IntroductionIntroduction

• Data Transport Challenges & Data Transport Challenges & SolutionsSolutions

• Rendering Challenges & SolutionsRendering Challenges & Solutions

• Conclusions/FuturesConclusions/Futures

• IntroductionIntroduction

• Data Transport Challenges & Data Transport Challenges & SolutionsSolutions

• Rendering Challenges & SolutionsRendering Challenges & Solutions

• Conclusions/FuturesConclusions/Futures

11

Data Transport ChallengesData Transport Challenges

• Reliable TransportReliable Transport– Low LatencyLow Latency

– Synchronized Arrival of Multi-StreamsSynchronized Arrival of Multi-Streams

• Large Bandwidth RequirementsLarge Bandwidth Requirements

• Reliable TransportReliable Transport– Low LatencyLow Latency

– Synchronized Arrival of Multi-StreamsSynchronized Arrival of Multi-Streams

• Large Bandwidth RequirementsLarge Bandwidth Requirements

No. Streams No. Streams & Resolution& Resolution

2D 2D Bandwidth Bandwidth

2Hz@24bpp, 2Hz@24bpp, MbpsMbps

3D 3D Bandwidth Bandwidth

2Hz@40bpp, 2Hz@40bpp, Mbps Mbps

5@320x2405@320x240 18.418.4 30.830.8

5@640x4805@640x480 73.873.8 122.8122.8

15@640x48015@640x480 221.4221.4 368.4368.4

12

Compression OpportunitiesCompression Opportunities

• Color Image CompressionColor Image Compression– Spatial Compression (JPEG)Spatial Compression (JPEG)

– Temporal Compression (MPEG)Temporal Compression (MPEG)

• Depth ImageDepth Image– 16-bit 1/z image16-bit 1/z image

• Elimination of Redundant DataElimination of Redundant Data – Overlapping FOVs from Multiple CamerasOverlapping FOVs from Multiple Cameras

– Reduces BW and Rendering Requirements !!Reduces BW and Rendering Requirements !!

• Color Image CompressionColor Image Compression– Spatial Compression (JPEG)Spatial Compression (JPEG)

– Temporal Compression (MPEG)Temporal Compression (MPEG)

• Depth ImageDepth Image– 16-bit 1/z image16-bit 1/z image

• Elimination of Redundant DataElimination of Redundant Data – Overlapping FOVs from Multiple CamerasOverlapping FOVs from Multiple Cameras

– Reduces BW and Rendering Requirements !!Reduces BW and Rendering Requirements !!

13

Overlapping Depth ImagesOverlapping Depth Images

14

Redundant Data Redundant Data EliminationElimination

• Algorithm: Compare Every Depth Algorithm: Compare Every Depth StreamStream

• ProsPros– Best compressionBest compression

– Eliminates all redundant DataEliminates all redundant Data

• ConsCons– Jeopardizes Real-Time PerformanceJeopardizes Real-Time Performance

– Not scalable with number of 3D camerasNot scalable with number of 3D cameras

– Maximizes network BW requirements between Maximizes network BW requirements between ‘3D Cameras’ ‘3D Cameras’

• Algorithm: Compare Every Depth Algorithm: Compare Every Depth StreamStream

• ProsPros– Best compressionBest compression

– Eliminates all redundant DataEliminates all redundant Data

• ConsCons– Jeopardizes Real-Time PerformanceJeopardizes Real-Time Performance

– Not scalable with number of 3D camerasNot scalable with number of 3D cameras

– Maximizes network BW requirements between Maximizes network BW requirements between ‘3D Cameras’ ‘3D Cameras’

15

Group-Based, Differential Group-Based, Differential Stream Compression Stream Compression

• Two Stream Differential Two Stream Differential ComparisonComparison– Maintains real-time performanceMaintains real-time performance

- Scales with number of camerasScales with number of cameras

- Static, Disjoint Stream GroupsStatic, Disjoint Stream Groups- Geometric coherency metricGeometric coherency metric

• One Reference Stream per GroupOne Reference Stream per Group

• Main Reference StreamMain Reference Stream– Stream most closely aligned with remote Stream most closely aligned with remote

viewpointviewpoint

– UncompressedUncompressed

• Two Stream Differential Two Stream Differential ComparisonComparison– Maintains real-time performanceMaintains real-time performance

- Scales with number of camerasScales with number of cameras

- Static, Disjoint Stream GroupsStatic, Disjoint Stream Groups- Geometric coherency metricGeometric coherency metric

• One Reference Stream per GroupOne Reference Stream per Group

• Main Reference StreamMain Reference Stream– Stream most closely aligned with remote Stream most closely aligned with remote

viewpointviewpoint

– UncompressedUncompressed

16

Grouping & Differential Grouping & Differential EncodingEncoding

Group1

Group2

Group3

6

8 9

2 3

7

1

4 5

Group2 5

17

Algorithm PerformanceAlgorithm Performance

• Using Synthetic Camera ArraysUsing Synthetic Camera Arrays– 13 and 22 3D-camera arrays13 and 22 3D-camera arrays

– Real-world 3D dataReal-world 3D data

• 5X Compression Ratio5X Compression Ratio

• 50-60% of Best Possible Point 50-60% of Best Possible Point ReductionReduction

• 10z Performance at 640x48010z Performance at 640x480

• Using Synthetic Camera ArraysUsing Synthetic Camera Arrays– 13 and 22 3D-camera arrays13 and 22 3D-camera arrays

– Real-world 3D dataReal-world 3D data

• 5X Compression Ratio5X Compression Ratio

• 50-60% of Best Possible Point 50-60% of Best Possible Point ReductionReduction

• 10z Performance at 640x48010z Performance at 640x480

18

Multi-Stream TransportMulti-Stream Transport

• Reliable is Good, so TCP/IPReliable is Good, so TCP/IP

WRONG!WRONG!• LatencyLatency

• Unsynchronized Arrival of Unsynchronized Arrival of StreamsStreams– Synchronized at CamerasSynchronized at Cameras

– Must be re-synchronize at rendererMust be re-synchronize at renderer

• TCP Flows Competing for BWTCP Flows Competing for BW

• Reliable is Good, so TCP/IPReliable is Good, so TCP/IP

WRONG!WRONG!• LatencyLatency

• Unsynchronized Arrival of Unsynchronized Arrival of StreamsStreams– Synchronized at CamerasSynchronized at Cameras

– Must be re-synchronize at rendererMust be re-synchronize at renderer

• TCP Flows Competing for BWTCP Flows Competing for BW

19

Frame Arrival Time Frame Arrival Time Variation Variation

20

Throughput Variation Throughput Variation

21

CP-RUDP CP-RUDP

• Reliable UDP (RUDP)Reliable UDP (RUDP)

• Coordination Protocol (CP)Coordination Protocol (CP)

• CharacteristicsCharacteristics– A TCP-friendly UDP protocolA TCP-friendly UDP protocol

– Aggregately acts like N TCP flowsAggregately acts like N TCP flows

– Providing BW coordination and synchronization of multiple Providing BW coordination and synchronization of multiple flowsflows

– Prevents send rates from exceeding network capacityPrevents send rates from exceeding network capacity

Flows that are ahead are given less BWFlows that are ahead are given less BW

Flows that are behind are given more Flows that are behind are given more BWBW

• Reliable UDP (RUDP)Reliable UDP (RUDP)

• Coordination Protocol (CP)Coordination Protocol (CP)

• CharacteristicsCharacteristics– A TCP-friendly UDP protocolA TCP-friendly UDP protocol

– Aggregately acts like N TCP flowsAggregately acts like N TCP flows

– Providing BW coordination and synchronization of multiple Providing BW coordination and synchronization of multiple flowsflows

– Prevents send rates from exceeding network capacityPrevents send rates from exceeding network capacity

Flows that are ahead are given less BWFlows that are ahead are given less BW

Flows that are behind are given more Flows that are behind are given more BWBW

22

Future CP-RUDP Testbed Future CP-RUDP Testbed

• Future Network Router FunctionalityFuture Network Router Functionality

• Emulated today on fast FreeBSD PCEmulated today on fast FreeBSD PC

• Future Network Router FunctionalityFuture Network Router Functionality

• Emulated today on fast FreeBSD PCEmulated today on fast FreeBSD PC

3D Camera Internet2

3D Camera

3D Camera

3D Rendering

User Tracker

CP-RUDPSoftware

Router

CP-RUDPSoftware

Router

23

Outline Outline

• IntroductionIntroduction

• Data Transport Challenges & Data Transport Challenges & SolutionsSolutions

• Rendering Challenges & SolutionsRendering Challenges & Solutions

• Conclusions/FuturesConclusions/Futures

• IntroductionIntroduction

• Data Transport Challenges & Data Transport Challenges & SolutionsSolutions

• Rendering Challenges & SolutionsRendering Challenges & Solutions

• Conclusions/FuturesConclusions/Futures

24

Rendering ChallengesRendering Challenges

• High Performance, Interactive High Performance, Interactive ArchitectureArchitecture

• Quality Surface RepresentationQuality Surface Representation

• Scalable PerformanceScalable Performance

• High Performance, Interactive High Performance, Interactive ArchitectureArchitecture

• Quality Surface RepresentationQuality Surface Representation

• Scalable PerformanceScalable Performance

25

High Performance, Interactive High Performance, Interactive ArchitectureArchitecture

• Frame Re-synchronizationFrame Re-synchronization– ‘‘N’ 3D-camera StreamsN’ 3D-camera Streams

– Timestamps on each data frameTimestamps on each data frame

– Buffering -> More Latency Buffering -> More Latency BAD!BAD!

• Multi-threaded Design for Multi-threaded Design for InteractivityInteractivity– 30Hz Refresh Rate Desirable30Hz Refresh Rate Desirable

– <10 Hz Data Update Rate<10 Hz Data Update Rate

• 3-PC Rendering Cluster (Linux)3-PC Rendering Cluster (Linux)– Left & right eye rendering nodesLeft & right eye rendering nodes

– 33rdrd Node – Network Aggregation Point Node – Network Aggregation Point

• Frame Re-synchronizationFrame Re-synchronization– ‘‘N’ 3D-camera StreamsN’ 3D-camera Streams

– Timestamps on each data frameTimestamps on each data frame

– Buffering -> More Latency Buffering -> More Latency BAD!BAD!

• Multi-threaded Design for Multi-threaded Design for InteractivityInteractivity– 30Hz Refresh Rate Desirable30Hz Refresh Rate Desirable

– <10 Hz Data Update Rate<10 Hz Data Update Rate

• 3-PC Rendering Cluster (Linux)3-PC Rendering Cluster (Linux)– Left & right eye rendering nodesLeft & right eye rendering nodes

– 33rdrd Node – Network Aggregation Point Node – Network Aggregation Point

26

Rendering Architecture Rendering Architecture

27

Quality Surface Quality Surface RepresentationRepresentation

• Depth Images (1/z + RGB per pixel)Depth Images (1/z + RGB per pixel)

• Real-Time TriangulationReal-Time Triangulation– Performance LimitingPerformance Limiting

– Artifacts due to noisy outliers. Artifacts due to noisy outliers. Most Disturbing!Most Disturbing!

• Real-Time Massive Point Cloud Real-Time Massive Point Cloud RenderingRendering– Display List vs. VARDisplay List vs. VAR

– GL_POINT vs. GL_POINT_SPRINTGL_POINT vs. GL_POINT_SPRINT

– Screen-Oriented vs. Object-Oriented SplatsScreen-Oriented vs. Object-Oriented Splats

• Depth Images (1/z + RGB per pixel)Depth Images (1/z + RGB per pixel)

• Real-Time TriangulationReal-Time Triangulation– Performance LimitingPerformance Limiting

– Artifacts due to noisy outliers. Artifacts due to noisy outliers. Most Disturbing!Most Disturbing!

• Real-Time Massive Point Cloud Real-Time Massive Point Cloud RenderingRendering– Display List vs. VARDisplay List vs. VAR

– GL_POINT vs. GL_POINT_SPRINTGL_POINT vs. GL_POINT_SPRINT

– Screen-Oriented vs. Object-Oriented SplatsScreen-Oriented vs. Object-Oriented Splats

28

Fixed-Size SplatsFixed-Size Splats

29

Screen vs. Object Space Screen vs. Object Space SplatsSplats

30

Scalable PerformanceScalable Performance

• 2x Gain w/3-PC Rendering Cluster2x Gain w/3-PC Rendering Cluster

• Ride the Wave (Moore’s Law & Ride the Wave (Moore’s Law & Beyond)Beyond)

Not Enough!Not Enough!

• Divide and ConquerDivide and Conquer– Eliminate Data Replication RequirementEliminate Data Replication Requirement

– Match # Primitives to PerformanceMatch # Primitives to Performance

• Depth Compositing Depth Compositing – UNC PixelFlow, Stanford/Intel Lightning-2 or HP SepiaUNC PixelFlow, Stanford/Intel Lightning-2 or HP Sepia

– DVI output – RGB and ZDVI output – RGB and Z

• 2x Gain w/3-PC Rendering Cluster2x Gain w/3-PC Rendering Cluster

• Ride the Wave (Moore’s Law & Ride the Wave (Moore’s Law & Beyond)Beyond)

Not Enough!Not Enough!

• Divide and ConquerDivide and Conquer– Eliminate Data Replication RequirementEliminate Data Replication Requirement

– Match # Primitives to PerformanceMatch # Primitives to Performance

• Depth Compositing Depth Compositing – UNC PixelFlow, Stanford/Intel Lightning-2 or HP SepiaUNC PixelFlow, Stanford/Intel Lightning-2 or HP Sepia

– DVI output – RGB and ZDVI output – RGB and Z

31

Conclusions/FuturesConclusions/Futures

• Much work remains !Much work remains !– Performance and FidelityPerformance and Fidelity

• Investigating Temporal Compression Investigating Temporal Compression of 3D Streamsof 3D Streams

• Testing CP-RUDP Network ProtocolTesting CP-RUDP Network Protocol

• New Voxel-Based SolutionNew Voxel-Based Solution– Elminates Redundant DataElminates Redundant Data

– Normals and Confidence Value per VoxelNormals and Confidence Value per Voxel

– GPU Programs for Object-oriented Quad Splats with GPU Programs for Object-oriented Quad Splats with Projective TextureProjective Texture

• Much work remains !Much work remains !– Performance and FidelityPerformance and Fidelity

• Investigating Temporal Compression Investigating Temporal Compression of 3D Streamsof 3D Streams

• Testing CP-RUDP Network ProtocolTesting CP-RUDP Network Protocol

• New Voxel-Based SolutionNew Voxel-Based Solution– Elminates Redundant DataElminates Redundant Data

– Normals and Confidence Value per VoxelNormals and Confidence Value per Voxel

– GPU Programs for Object-oriented Quad Splats with GPU Programs for Object-oriented Quad Splats with Projective TextureProjective Texture

32

Research Sponsors Research Sponsors

• National Science FoundationNational Science Foundation– Grant IIS-0121293 Junku Yuh, Program DirectorGrant IIS-0121293 Junku Yuh, Program Director

• DARPADARPA– 3-D Tele-Immersion over NGI3-D Tele-Immersion over NGI

• Advanced Network & Services, Advanced Network & Services, Armonk, NYArmonk, NY– National Tele-Immersion Initiative (NTII), 1998-2000National Tele-Immersion Initiative (NTII), 1998-2000

• National Science FoundationNational Science Foundation– Grant IIS-0121293 Junku Yuh, Program DirectorGrant IIS-0121293 Junku Yuh, Program Director

• DARPADARPA– 3-D Tele-Immersion over NGI3-D Tele-Immersion over NGI

• Advanced Network & Services, Advanced Network & Services, Armonk, NYArmonk, NY– National Tele-Immersion Initiative (NTII), 1998-2000National Tele-Immersion Initiative (NTII), 1998-2000

33

Thank You Thank You

UNC - January 2000

34

Thank You Thank You

UNC - January 2000

35

Positive Technology TrendsPositive Technology Trends

• Low-cost image sensorsLow-cost image sensors

• Vigorous research into new light field and Vigorous research into new light field and image-based scene reconstruction image-based scene reconstruction algorithmsalgorithms

• Incredible performance in commodity PCs Incredible performance in commodity PCs and 3D graphicsand 3D graphics

• Ease of Developing Custom HardwareEase of Developing Custom Hardware

• New Displays – HiRes LCD and Plasma New Displays – HiRes LCD and Plasma panels, Tiled-Projectors, OLEDs, Auto-panels, Tiled-Projectors, OLEDs, Auto-stereoscopicstereoscopic

• Increasing network bandwidth to the Increasing network bandwidth to the workplace and homeworkplace and home

• Low-cost image sensorsLow-cost image sensors

• Vigorous research into new light field and Vigorous research into new light field and image-based scene reconstruction image-based scene reconstruction algorithmsalgorithms

• Incredible performance in commodity PCs Incredible performance in commodity PCs and 3D graphicsand 3D graphics

• Ease of Developing Custom HardwareEase of Developing Custom Hardware

• New Displays – HiRes LCD and Plasma New Displays – HiRes LCD and Plasma panels, Tiled-Projectors, OLEDs, Auto-panels, Tiled-Projectors, OLEDs, Auto-stereoscopicstereoscopic

• Increasing network bandwidth to the Increasing network bandwidth to the workplace and homeworkplace and home

36

Renderer Block Diagram Renderer Block Diagram

37

• Information Society Technologies (IST) Programme of the Information Society Technologies (IST) Programme of the European CommissionEuropean Commission

http://bs.hhi.de/projects/VIRTUE.htmhttp://bs.hhi.de/projects/VIRTUE.htm

• Information Society Technologies (IST) Programme of the Information Society Technologies (IST) Programme of the European CommissionEuropean Commission

http://bs.hhi.de/projects/VIRTUE.htmhttp://bs.hhi.de/projects/VIRTUE.htm

VIRTUE: VIRtual Team User VIRTUE: VIRtual Team User EnvironmentEnvironment

38

VIRTUE PartnersVIRTUE Partners

• Heinrich-Hertz Institut (now Heinrich-Hertz Institut (now Fraunhofer member)Fraunhofer member)– Disparity Estimator, Real-time ComponentsDisparity Estimator, Real-time Components

• University of DelftUniversity of Delft– 3D Algorithms, Image-Based Rendering3D Algorithms, Image-Based Rendering

• TNO Human FactorsTNO Human Factors• Sony U.K. Sony U.K.

– System design, Compositor, Rendering, MPEG-2System design, Compositor, Rendering, MPEG-2

• Harriot Watt UniversityHarriot Watt University– 3D Video Processing3D Video Processing

• British Telecom British Telecom – Project Prime, Video-based Head TrackingProject Prime, Video-based Head Tracking

• Heinrich-Hertz Institut (now Heinrich-Hertz Institut (now Fraunhofer member)Fraunhofer member)– Disparity Estimator, Real-time ComponentsDisparity Estimator, Real-time Components

• University of DelftUniversity of Delft– 3D Algorithms, Image-Based Rendering3D Algorithms, Image-Based Rendering

• TNO Human FactorsTNO Human Factors• Sony U.K. Sony U.K.

– System design, Compositor, Rendering, MPEG-2System design, Compositor, Rendering, MPEG-2

• Harriot Watt UniversityHarriot Watt University– 3D Video Processing3D Video Processing

• British Telecom British Telecom – Project Prime, Video-based Head TrackingProject Prime, Video-based Head Tracking

39

VIRTUE TechnologyVIRTUE Technology

• Realistic Wide View Synthesis for Realistic Wide View Synthesis for Dynamic ScenesDynamic Scenes

• Object-based SegmentationObject-based Segmentation

• Disparity Engine runs 4x4 grid @ Disparity Engine runs 4x4 grid @ videoratevideorate

• Facial feature (eye) trackingFacial feature (eye) tracking

• Generating View-dependent Novel ViewGenerating View-dependent Novel View

• 2D Display (with motion parallax) 2D Display (with motion parallax) rendered with virtual backgroundrendered with virtual background

Demonstrated at IBC – Fall 2002Demonstrated at IBC – Fall 2002

• Realistic Wide View Synthesis for Realistic Wide View Synthesis for Dynamic ScenesDynamic Scenes

• Object-based SegmentationObject-based Segmentation

• Disparity Engine runs 4x4 grid @ Disparity Engine runs 4x4 grid @ videoratevideorate

• Facial feature (eye) trackingFacial feature (eye) tracking

• Generating View-dependent Novel ViewGenerating View-dependent Novel View

• 2D Display (with motion parallax) 2D Display (with motion parallax) rendered with virtual backgroundrendered with virtual background

Demonstrated at IBC – Fall 2002Demonstrated at IBC – Fall 2002

40

Image-based Visual Image-based Visual HullHull

Image-based Visual Image-based Visual HullHull

HP Labs ‘Coliseum’ System HP Labs ‘Coliseum’ System

41

HP Labs ‘Coliseum’ SystemHP Labs ‘Coliseum’ System

• Personal Immersive Conferencing Personal Immersive Conferencing

• Based on ‘Visual Hull’ research of Based on ‘Visual Hull’ research of Matusik, McMillan et al. at MITMatusik, McMillan et al. at MIT– Space carving algorithmSpace carving algorithm

– Foreground/Background Foreground/Background

• 5 camera (1394) array, Now runs on 1 5 camera (1394) array, Now runs on 1 PCPC– 224 x 224 @ 10 Hz224 x 224 @ 10 Hz

• 300 x 300 display window300 x 300 display window– 3D Volume Composited with 3D VRML scene3D Volume Composited with 3D VRML scene

Demonstrated at ITP – Dec 2002Demonstrated at ITP – Dec 2002

• Personal Immersive Conferencing Personal Immersive Conferencing

• Based on ‘Visual Hull’ research of Based on ‘Visual Hull’ research of Matusik, McMillan et al. at MITMatusik, McMillan et al. at MIT– Space carving algorithmSpace carving algorithm

– Foreground/Background Foreground/Background

• 5 camera (1394) array, Now runs on 1 5 camera (1394) array, Now runs on 1 PCPC– 224 x 224 @ 10 Hz224 x 224 @ 10 Hz

• 300 x 300 display window300 x 300 display window– 3D Volume Composited with 3D VRML scene3D Volume Composited with 3D VRML scene

Demonstrated at ITP – Dec 2002Demonstrated at ITP – Dec 2002

42

CreditsCredits

• Colleagues at Colleagues at – UPenn: Kostas Daniilidis (Co-PI), Nikhil Kelshikar, UPenn: Kostas Daniilidis (Co-PI), Nikhil Kelshikar,

Xenofon ZabulisXenofon Zabulis

– PSC: John Urbanic, Kathy BenningerPSC: John Urbanic, Kathy Benninger

– Advanced Networks & Services: Jaron Lanier, Amela Advanced Networks & Services: Jaron Lanier, Amela SadagicSadagic

– UNC: Henry Fuchs (PI), Greg Welch, Ketan Mayer-PatelUNC: Henry Fuchs (PI), Greg Welch, Ketan Mayer-Patel

• UNC RAsUNC RAs– Ruigang Yang, Wei-Chao Chen, Sang-Uok Kum, Sudipta Ruigang Yang, Wei-Chao Chen, Sang-Uok Kum, Sudipta

Sinha, Scott Larsen, Vivek Sawant, Travis Sparks, David Sinha, Scott Larsen, Vivek Sawant, Travis Sparks, David Ott, and Gabe SuOtt, and Gabe Su

• Colleagues at Colleagues at – UPenn: Kostas Daniilidis (Co-PI), Nikhil Kelshikar, UPenn: Kostas Daniilidis (Co-PI), Nikhil Kelshikar,

Xenofon ZabulisXenofon Zabulis

– PSC: John Urbanic, Kathy BenningerPSC: John Urbanic, Kathy Benninger

– Advanced Networks & Services: Jaron Lanier, Amela Advanced Networks & Services: Jaron Lanier, Amela SadagicSadagic

– UNC: Henry Fuchs (PI), Greg Welch, Ketan Mayer-PatelUNC: Henry Fuchs (PI), Greg Welch, Ketan Mayer-Patel

• UNC RAsUNC RAs– Ruigang Yang, Wei-Chao Chen, Sang-Uok Kum, Sudipta Ruigang Yang, Wei-Chao Chen, Sang-Uok Kum, Sudipta

Sinha, Scott Larsen, Vivek Sawant, Travis Sparks, David Sinha, Scott Larsen, Vivek Sawant, Travis Sparks, David Ott, and Gabe SuOtt, and Gabe Su

43

Our Vision: ‘Xtreme Tele-Our Vision: ‘Xtreme Tele-PresencePresence

‘Office of the Future’

Andrei State 1998

44

Our Vision: ‘Xtreme Our Vision: ‘Xtreme ChallengesChallenges

‘Office of the Future’

Andrei State 1998

Distributed Graphics• Collaborative Datasets• User Interfaces

3D Scene Acquisition• Cameras• 3D Reconstruction

Presentation• 3D Stereo Display• View Dependent• Front Projection• Spatialized Audio

Networking• QoS • BW-Latency-Jitter

45

Immersive Electronic BooksImmersive Electronic Books

• New Paradigm for Surgical TrainingNew Paradigm for Surgical Training

• Allow Surgeons to Witness & ExploreAllow Surgeons to Witness & Explore– Replay a surgical procedure from any novel viewpoint in Replay a surgical procedure from any novel viewpoint in

3D3D

– Augmented with annotations & relevant medical Augmented with annotations & relevant medical metadatametadata

• New Paradigm for Surgical TrainingNew Paradigm for Surgical Training

• Allow Surgeons to Witness & ExploreAllow Surgeons to Witness & Explore– Replay a surgical procedure from any novel viewpoint in Replay a surgical procedure from any novel viewpoint in

3D3D

– Augmented with annotations & relevant medical Augmented with annotations & relevant medical metadatametadata

46

Remote Medical Remote Medical ConsultationConsultation