ibvh

8/8/2019 ibvh

1/65

Image-Based Visual HullsImage-Based Visual Hulls

Wojciech MatusikWojciech Matusik

Chris BuehlerChris Buehler

Leonard McMillanLeonard McMillan

Massachusetts Institute of Technology

Laboratory for Computer Science

Ramesh RaskarRamesh RaskarUniversity of North Carolina

at Chapel Hill

Steven J. GortlerSteven J. GortlerHarvard University

8/8/2019 ibvh

2/65

MotivationMotivation

RealReal--time acquisition and rendering of dynamic scenestime acquisition and rendering of dynamic scenes

8/8/2019 ibvh

3/65

Previous WorkPrevious Work

Virtualized RealityVirtualized Reality(Rander97, Kanade97, Narayanan98)(Rander97, Kanade97, Narayanan98)

Visual HullVisual Hull(Laurentini94)(Laurentini94)

Volume CarvingVolume Carving(Potmesil87, Szeliski93, Seitz97)(Potmesil87, Szeliski93, Seitz97)

CSG RenderingCSG Rendering(Goldfeather86, Rappoport97)(Goldfeather86, Rappoport97) ImageImage--Based RenderingBased Rendering

(McMillan95, Debevec96, Debevec98)(McMillan95, Debevec96, Debevec98)

8/8/2019 ibvh

4/65

ContributionsContributions

ViewView--dependent imagedependent image--basedbasedvisual hull representationvisual hull representation

Efficient algorithm for rendering theEfficient algorithm for rendering thevisual hullvisual hull Efficient algorithm computing visibilityEfficient algorithm computing visibility

A realA real--time systemtime system

8/8/2019 ibvh

5/65

What is a Visual Hull?What is a Visual Hull?

8/8/2019 ibvh

6/65

Why use a Visual Hull?Why use a Visual Hull?

Can be computed robustlyCan be computed robustly Can be computed efficientlyCan be computed efficiently

- =

backgroundbackground

++

foregroundforeground

backgroundbackground foregroundforeground

8/8/2019 ibvh

7/65

Rendering Visual HullsRendering Visual Hulls

Reference 1

Reference 2

Desired

8/8/2019 ibvh

8/65

Build then SampleBuild then Sample

Reference 1

Reference 2

Desired

8/8/2019 ibvh

9/65


Reference 1

Reference 2

Desired

8/8/2019 ibvh

10/65


Reference 1

Reference 2

Desired

8/8/2019 ibvh

11/65


Reference 1

Reference 2

Desired

8/8/2019 ibvh

12/65


Reference 1

Reference 2

Desired

8/8/2019 ibvh

13/65

Sample DirectlySample Directly

Reference 1

Reference 2

Desired

8/8/2019 ibvh

14/65


Reference 1

Reference 2

Desired

8/8/2019 ibvh

15/65


Reference 1

Reference 2

Desired

8/8/2019 ibvh

16/65


Reference 1

Reference 2

Desired

8/8/2019 ibvh

17/65


Reference 1

Reference 2

Desired

8/8/2019 ibvh

18/65


Reference 1

Reference 2

Desired

8/8/2019 ibvh

19/65


Reference 1

Reference 2

Desired

8/8/2019 ibvh

20/65


Reference 1

Reference 2

Desired

8/8/2019 ibvh

21/65

Direct Sampling AdvantagesDirect Sampling Advantages

Line interval intersections are robustLine interval intersections are robust Direct sampling gives us exact renderingDirect sampling gives us exact rendering

Can be computed efficiently inCan be computed efficiently inimage spaceimage space

8/8/2019 ibvh

22/65

Image-Based ComputationImage-Based Computation

Reference 1

Reference 2

Desired

8/8/2019 ibvh

23/65

ObservationObservation

Incremental computation along scanlinesIncremental computation along scanlines

Desired

Reference

8/8/2019 ibvh

24/65

BinningBinning

Epipole

Sort silhouette edges into binsSort silhouette edges into bins

8/8/2019 ibvh

25/65

BinningBinning

Epipole


8/8/2019 ibvh

26/65

BinningBinning


Epipole

Bin 1

8/8/2019 ibvh

27/65

BinningBinning


Epipole

Bin 2

Bin 1

8/8/2019 ibvh

28/65

BinningBinning


Epipole

Bin 3

Bin 1

Bin 2

8/8/2019 ibvh

29/65

BinningBinning


Epipole

Bin 4

Bin 1

Bin 2

Bin 3

8/8/2019 ibvh

30/65

BinningBinning


Epipole

Bin 5

Bin 1

Bin 2

Bin 3

Bin 4

8/8/2019 ibvh

31/65

BinningBinning


Epipole

Bin 5

Bin 1

Bin 2

Bin 3

Bin 4

8/8/2019 ibvh

32/65

ScanningScanning

Epipole

Bin 1

8/8/2019 ibvh

33/65

Epipole

Bin 2

ScanningScanning

8/8/2019 ibvh

34/65

Epipole

Bin 2

ScanningScanning

8/8/2019 ibvh

35/65

Epipole

Bin 2

ScanningScanning

8/8/2019 ibvh

36/65

Epipole

Bin 4

ScanningScanning

8/8/2019 ibvh

37/65

Epipole

Bin 5

ScanningScanning

8/8/2019 ibvh

38/65

Coarse-to-Fine SamplingCoarse-to-Fine Sampling

8/8/2019 ibvh

39/65

IBVH ResultsIBVH Results

Approximately constant computation perApproximately constant computation perpixel per camerapixel per camera

ParallelizesParallelizes Consistent withConsistent with

input silhouettesinput silhouettes

8/8/2019 ibvh

40/65

Video of IBVHVideo of IBVH

8/8/2019 ibvh

41/65

Shading AlgorithmShading Algorithm

A viewA view--dependent strategydependent strategy

8/8/2019 ibvh

42/65

Visibility AlgorithmVisibility Algorithm

8/8/2019 ibvh

43/65

Visibility in 2DVisibility in 2D

Desired viewReference view

8/8/2019 ibvh

44/65



Front-most Points

8/8/2019 ibvh

45/65



Visible

8/8/2019 ibvh

46/65



Coverage Mask

8/8/2019 ibvh

47/65



Coverage Mask

Visible

8/8/2019 ibvh

48/65



Coverage Mask

Visible

8/8/2019 ibvh

49/65



Coverage Mask

VisibleNot

8/8/2019 ibvh

50/65



Coverage Mask

8/8/2019 ibvh

51/65



Coverage Mask

Visible

8/8/2019 ibvh

52/65



Coverage Mask

8/8/2019 ibvh

53/65



Coverage Mask

VisibleNot

8/8/2019 ibvh

54/65

Shaded Visual HullsShaded Visual Hulls

8/8/2019 ibvh

55/65

SystemSystem

Server

(4x 500 Mhz)

Camera

Client

Camera

Client

Camera

Client

Camera

Client

8/8/2019 ibvh

56/65

SystemSystem

Server

(4x 500 Mhz)

Camera

Client

Camera

Client

Camera

Client

Camera

Client

TriggerSignal

8/8/2019 ibvh

57/65

SystemSystem

Server

(4x 500 Mhz)

Camera

Client

Camera

Client

Camera

Client

Camera

Client

8/8/2019 ibvh

58/65

SystemSystem

Server

(4x 500 Mhz)

Camera

Client

Camera

Client

Camera

Client

Camera

Client

Compressed video

8/8/2019 ibvh

59/65

SystemSystem

Server

(4x 500 Mhz)

Camera

Client

Camera

Client

Camera

Client

Camera

Client

Intersection

8/8/2019 ibvh

60/65

SystemSystem

Server

(4x 500 Mhz)

Camera

Client

Camera

Client

Camera

Client

Camera

Client

Visibility

8/8/2019 ibvh

61/65

SystemSystem

Server

(4x 500 Mhz)

Camera

Client

Camera

Client

Camera

Client

Camera

Client

Shading

8/8/2019 ibvh

62/65

More IBVH ResultsMore IBVH Results

8/8/2019 ibvh

63/65

Future WorkFuture Work

3D teleconferencing3D teleconferencing Distributed computationDistributed computation Virtual setsVirtual sets PostPost--production camera effectsproduction camera effects Mixed realityMixed reality

8/8/2019 ibvh

64/65

SummarySummary

Visual hulls with texture can provide aVisual hulls with texture can provide acompelling realcompelling real--time visualizationstime visualizations

Visual hulls can be computed accuratelyVisual hulls can be computed accuratelyand efficiently in image spaceand efficiently in image space

View dependent shading withView dependent shading withconservative visibilityconservative visibility

8/8/2019 ibvh

65/65

AcknowledgementsAcknowledgements

DARPA ITOGrant F30602DARPA ITOGrant F30602--971971--02830283 A generous grant from Intel CorporationA generous grant from Intel Corporation NSF Career Awards 9875859 & 9703399NSF Career Awards 9875859 & 9703399 Tom Buehler & Kari Anne KjlassTom Buehler & Kari Anne Kjlass

Thanks to all members of the

MIT Computer Graphics Group

Documents

ibvh