42
Visibility Sorting and Visibility Sorting and Compositing without Compositing without Splitting Splitting for Image Layer for Image Layer Decompositions Decompositions John Snyder, Jed Lengyel John Snyder, Jed Lengyel

Visibility Sorting and Compositing without Splitting for Image Layer Decompositions

Embed Size (px)

DESCRIPTION

Visibility Sorting and Compositing without Splitting for Image Layer Decompositions. John Snyder, Jed Lengyel. Layered Decomposition Problem. Given set of moving geometric parts , and moving camera, find visibility ordering for every frame. sort. composite. parts. sorted images. - PowerPoint PPT Presentation

Citation preview

Visibility Sorting and Compositing Visibility Sorting and Compositing without Splitting without Splitting

for Image Layer Decompositionsfor Image Layer Decompositions

Visibility Sorting and Compositing Visibility Sorting and Compositing without Splitting without Splitting

for Image Layer Decompositionsfor Image Layer Decompositions

John Snyder, Jed LengyelJohn Snyder, Jed LengyelJohn Snyder, Jed LengyelJohn Snyder, Jed Lengyel

Layered Decomposition ProblemLayered Decomposition ProblemLayered Decomposition ProblemLayered Decomposition Problem

Given set of moving geometric Given set of moving geometric partsparts, ,

and moving camera,and moving camera,

find visibility ordering for every frame.find visibility ordering for every frame.

Given set of moving geometric Given set of moving geometric partsparts, ,

and moving camera,and moving camera,

find visibility ordering for every frame.find visibility ordering for every frame.

partsparts sorted imagessorted images composited resultcomposited result

sortsort compositecomposite

Occlusion CyclesOcclusion CyclesOcclusion CyclesOcclusion Cycles

C

B

A C

B

A

When no order exists, can split or group parts.When no order exists, can split or group parts.

C2

C1

B

A C2

C1

B

A

G

G

G G

G

G

splitsplit

groupgroup

Non-SplittingNon-Splitting Layered Decomposition Layered DecompositionNon-SplittingNon-Splitting Layered Decomposition Layered Decomposition

Don’t split parts that form occlusion cycles:Don’t split parts that form occlusion cycles:

• run-time splitting is slowrun-time splitting is slow

• splitting is often unnecessarysplitting is often unnecessary

• splitting destroys coherencesplitting destroys coherence

Don’t split parts that form occlusion cycles:Don’t split parts that form occlusion cycles:

• run-time splitting is slowrun-time splitting is slow

• splitting is often unnecessarysplitting is often unnecessary

• splitting destroys coherencesplitting destroys coherence

Sorting requires no global separating plane.Sorting requires no global separating plane.

Image-Based Rendering AccelerationImage-Based Rendering Acceleration[Lengyel97][Lengyel97]Image-Based Rendering AccelerationImage-Based Rendering Acceleration[Lengyel97][Lengyel97]

spritessprites composited imagecomposited image

Other ApplicationsOther ApplicationsOther ApplicationsOther Applications

• fast special effects fast special effects

• animation playback with selective displayanimation playback with selective display

• incorporation of external image streamsincorporation of external image streams

• image stream compressionimage stream compression

• fast hidden line renderingfast hidden line rendering

• z resolution targetingz resolution targeting

• fast special effects fast special effects

• animation playback with selective displayanimation playback with selective display

• incorporation of external image streamsincorporation of external image streams

• image stream compressionimage stream compression

• fast hidden line renderingfast hidden line rendering

• z resolution targetingz resolution targeting

Previous WorkPrevious WorkPrevious WorkPrevious Work

• invisibility cullinginvisibility culling [Greene93,Zhang97] [Greene93,Zhang97]

• analytic hidden surface removalanalytic hidden surface removal [Mulmuley89] [Mulmuley89]

• global visibilityglobal visibility [Teller93,Durand97] [Teller93,Durand97]

• image layer decompositionsimage layer decompositions      [Schumacker69,Newell72,Fuchs80]     [Schumacker69,Newell72,Fuchs80]

• dynamic visibilitydynamic visibility [Torres90,Sudarsky96] [Torres90,Sudarsky96]

• depth sorting for special effectsdepth sorting for special effects [Max85] [Max85]

• invisibility cullinginvisibility culling [Greene93,Zhang97] [Greene93,Zhang97]

• analytic hidden surface removalanalytic hidden surface removal [Mulmuley89] [Mulmuley89]

• global visibilityglobal visibility [Teller93,Durand97] [Teller93,Durand97]

• image layer decompositionsimage layer decompositions      [Schumacker69,Newell72,Fuchs80]     [Schumacker69,Newell72,Fuchs80]

• dynamic visibilitydynamic visibility [Torres90,Sudarsky96] [Torres90,Sudarsky96]

• depth sorting for special effectsdepth sorting for special effects [Max85] [Max85]

Occlusion GraphsOcclusion GraphsOcclusion GraphsOcclusion Graphs

•pairwisepairwise occlusion relation: A occlusion relation: AEE B B

•relations on parts forms directed graph:    relations on parts forms directed graph:    occlusion occlusion graphgraph

•pairwisepairwise occlusion relation: A occlusion relation: AEE B B

•relations on parts forms directed graph:    relations on parts forms directed graph:    occlusion occlusion graphgraph

AB

view from Eview from E

AA BB

side viewside view

EE

Occlusion Graphs (Ex. 1)Occlusion Graphs (Ex. 1)Occlusion Graphs (Ex. 1)Occlusion Graphs (Ex. 1)

C

B

A A B

C

Occlusion Graphs (Ex. 2)Occlusion Graphs (Ex. 2)Occlusion Graphs (Ex. 2)Occlusion Graphs (Ex. 2)

C

B

A A B

C

Occlusion Graphs (Ex. 3)Occlusion Graphs (Ex. 3)Occlusion Graphs (Ex. 3)Occlusion Graphs (Ex. 3)

C

B

A C

B

A

A B

C

Forms strongly connected component (SCC).Forms strongly connected component (SCC).

Resolving Non-Binary CyclesResolving Non-Binary CyclesResolving Non-Binary CyclesResolving Non-Binary Cycles

SCCs need not be rendered as an aggregate!SCCs need not be rendered as an aggregate!

A B C

sum

A out C B out A C out B

A B C

sum

A out C B out A C out B

Occlusion Testing via CollisionOcclusion Testing via CollisionOcclusion Testing via CollisionOcclusion Testing via Collision

EE

B A

For convex hulls of objects A and B:

BE A ch(A E) B For convex hulls of objects A and B:

BE A ch(A E) B

Occlusion Testing via Collision:Occlusion Testing via Collision:Occluding ExampleOccluding ExampleOcclusion Testing via Collision:Occlusion Testing via Collision:Occluding ExampleOccluding Example

EE

ch(A E) B BE Ach(A E) B BE A

ch(A E)B

Occlusion Testing via Collision:Occlusion Testing via Collision:Non-Occluding ExampleNon-Occluding ExampleOcclusion Testing via Collision:Occlusion Testing via Collision:Non-Occluding ExampleNon-Occluding Example

EEch(A E) B

////ch(A E) B BE Ach(A E) B BE A

Trivial Sorting AlgorithmTrivial Sorting AlgorithmTrivial Sorting AlgorithmTrivial Sorting Algorithm

• compute occlusion graphcompute occlusion graph

• detect and topologically sort SCCsdetect and topologically sort SCCs

• at least quadratic in number of parts!at least quadratic in number of parts!

• compute occlusion graphcompute occlusion graph

• detect and topologically sort SCCsdetect and topologically sort SCCs

• at least quadratic in number of parts!at least quadratic in number of parts!

EEA B C D F

Improving the Trivial AlgorithmImproving the Trivial AlgorithmImproving the Trivial AlgorithmImproving the Trivial Algorithm

EEA

B CD F

A B,C,D,F but nothing occludes A. Process A first.A B,C,D,F but nothing occludes A. Process A first.

Improving the Trivial AlgorithmImproving the Trivial AlgorithmImproving the Trivial AlgorithmImproving the Trivial Algorithm

EE B CD F

Sorting Output: ASorting Output: A

A

Improving the Trivial AlgorithmImproving the Trivial AlgorithmImproving the Trivial AlgorithmImproving the Trivial Algorithm

EEC

D F

Sorting Output: A, BSorting Output: A, B

B

Improving the Trivial AlgorithmImproving the Trivial AlgorithmImproving the Trivial AlgorithmImproving the Trivial Algorithm

EE D F

Sorting Output: A, B, CSorting Output: A, B, C

C

Improving the Trivial AlgorithmImproving the Trivial AlgorithmImproving the Trivial AlgorithmImproving the Trivial Algorithm

EE

Sorting Output: A, B, C, D, FSorting Output: A, B, C, D, F

D F

Incremental Visibility Sort (IVS)Incremental Visibility Sort (IVS)Incremental Visibility Sort (IVS)Incremental Visibility Sort (IVS)

•similar to Newell, Newell, & Sancha algorithm, 1972similar to Newell, Newell, & Sancha algorithm, 1972

•sorts parts, not polygonssorts parts, not polygons

•detects SCCs, doesn’t split detects SCCs, doesn’t split

•uses order from last query, not depth orderuses order from last query, not depth order

•culls more efficiently using kd-treeculls more efficiently using kd-tree

•similar to Newell, Newell, & Sancha algorithm, 1972similar to Newell, Newell, & Sancha algorithm, 1972

•sorts parts, not polygonssorts parts, not polygons

•detects SCCs, doesn’t split detects SCCs, doesn’t split

•uses order from last query, not depth orderuses order from last query, not depth order

•culls more efficiently using kd-treeculls more efficiently using kd-tree

Depth order doesn’t Depth order doesn’t indicate visibility order.indicate visibility order.Depth order doesn’t Depth order doesn’t indicate visibility order.indicate visibility order.

EE

A

B

ZZ

IVS AlgorithmIVS AlgorithmIVS AlgorithmIVS Algorithm

While L is nonempty, pop off top element AWhile L is nonempty, pop off top element A

if A is unmarkedif A is unmarked

if nothing remaining on L occludes Aif nothing remaining on L occludes Asend A to outputsend A to output

elseelsemark and reinsert A into Lmark and reinsert A into L

else // A is markedelse // A is markedcheck for occlusion cyclecheck for occlusion cycle

if found, group cycle objects and reinsertif found, group cycle objects and reinsertelse reinsert A into Lelse reinsert A into L

While L is nonempty, pop off top element AWhile L is nonempty, pop off top element A

if A is unmarkedif A is unmarked

if nothing remaining on L occludes Aif nothing remaining on L occludes Asend A to outputsend A to output

elseelsemark and reinsert A into Lmark and reinsert A into L

else // A is markedelse // A is markedcheck for occlusion cyclecheck for occlusion cycle

if found, group cycle objects and reinsertif found, group cycle objects and reinsertelse reinsert A into Lelse reinsert A into L

Occlusion CullingOcclusion CullingOcclusion CullingOcclusion CullingFundamental IVS query: Fundamental IVS query:

Which parts remaining in L occlude AWhich parts remaining in L occlude A??

• use convex hulls around partsuse convex hulls around parts

• bound hulls with spatial and angular extentsbound hulls with spatial and angular extents

• occlusion cull reduces to 1D interval intersectionocclusion cull reduces to 1D interval intersection

• bounded parts organized in dynamic kd-treebounded parts organized in dynamic kd-tree

Fundamental IVS query: Fundamental IVS query:

Which parts remaining in L occlude AWhich parts remaining in L occlude A??

• use convex hulls around partsuse convex hulls around parts

• bound hulls with spatial and angular extentsbound hulls with spatial and angular extents

• occlusion cull reduces to 1D interval intersectionocclusion cull reduces to 1D interval intersection

• bounded parts organized in dynamic kd-treebounded parts organized in dynamic kd-tree

objectobject convex hullconvex hull kd extentkd extent

Spatial ExtentsSpatial ExtentsSpatial ExtentsSpatial Extents

////

D

EA

B

D

EA

BD

EB A

D

EB A

B B A AB B A A B B A AB B A A

Angular ExtentsAngular ExtentsAngular ExtentsAngular Extents

////

EA

B

Z

X

Y

EA

B

Z

X

Y

EA

B

Z

X

Y

EA

B

Z

X

Y

B B A AB B A A B B A AB B A A

Kd-Tree Occlusion CullingKd-Tree Occlusion CullingKd-Tree Occlusion CullingKd-Tree Occlusion Culling

• rebalance kd-tree every framerebalance kd-tree every frame

• kd-tree supports part deactivationkd-tree supports part deactivation

• rebalance kd-tree every framerebalance kd-tree every frame

• kd-tree supports part deactivationkd-tree supports part deactivation

B

DC

A

xx11

yy11 yy22

xx11

yy11yy22

A CBD

yyxx

Exact Occlusion TestingExact Occlusion TestingExact Occlusion TestingExact Occlusion Testing

Does Does B B A? A?

• uses hybrid Chung/Gilbert collision algorithmuses hybrid Chung/Gilbert collision algorithm

• requires extremal vertex query on A and Brequires extremal vertex query on A and B

• no need to create no need to create ch(Ach(A E) E) dynamically dynamically

• exploits coherence in object motionexploits coherence in object motion

Does Does B B A? A?

• uses hybrid Chung/Gilbert collision algorithmuses hybrid Chung/Gilbert collision algorithm

• requires extremal vertex query on A and Brequires extremal vertex query on A and B

• no need to create no need to create ch(Ach(A E) E) dynamically dynamically

• exploits coherence in object motionexploits coherence in object motion

DDextremal vertex minimizes D•v on Aextremal vertex minimizes D•v on AA vv

Results (Video)Results (Video)Results (Video)Results (Video)

IVS ComplexityIVS ComplexityIVS ComplexityIVS Complexity

(n+r)(s + c o + c )2 (n+r)(s + c o + c )2

variable “typical”case

worstcase

n - number of parts n n

r - number of reinsertions O(n) O(n2)

s - occluder search complexity O(log n) O(n)

c - maximum size of SCC O(1) O(n)

o - maximum num. of occluders O(1) O(n)

total O(n log n) O(n4)

variable “typical”case

worstcase

n - number of parts n n

r - number of reinsertions O(n) O(n2)

s - occluder search complexity O(log n) O(n)

c - maximum size of SCC O(1) O(n)

o - maximum num. of occluders O(1) O(n)

total O(n log n) O(n4)

Performance with Increasing ObjectsPerformance with Increasing ObjectsPerformance with Increasing ObjectsPerformance with Increasing Objects

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0 500 1000 1500 2000

number of objects

cpu

sec

on

ds

maxavg

Future WorkFuture WorkFuture WorkFuture Work

• animation previeweranimation previewer

• continuous time queriescontinuous time queries

• adaptive splitting, especially for terrainadaptive splitting, especially for terrain

• faster, less conservative visibility testsfaster, less conservative visibility tests

• automatic part decompositionautomatic part decomposition

• visibility sorting with minimal splittingvisibility sorting with minimal splitting

• animation previeweranimation previewer

• continuous time queriescontinuous time queries

• adaptive splitting, especially for terrainadaptive splitting, especially for terrain

• faster, less conservative visibility testsfaster, less conservative visibility tests

• automatic part decompositionautomatic part decomposition

• visibility sorting with minimal splittingvisibility sorting with minimal splitting

ConclusionsConclusionsConclusionsConclusions

• non-splitting layered decomposition useful non-splitting layered decomposition useful

• software visibility sorting is practical software visibility sorting is practical

• basic ideas of approach:basic ideas of approach:

• exploit temporal and spatial coherenceexploit temporal and spatial coherence

• exploit properties of convex objectsexploit properties of convex objects

• exploit incremental collision detection algorithmsexploit incremental collision detection algorithms

• non-splitting layered decomposition useful non-splitting layered decomposition useful

• software visibility sorting is practical software visibility sorting is practical

• basic ideas of approach:basic ideas of approach:

• exploit temporal and spatial coherenceexploit temporal and spatial coherence

• exploit properties of convex objectsexploit properties of convex objects

• exploit incremental collision detection algorithmsexploit incremental collision detection algorithms

Extent TrackingExtent TrackingExtent TrackingExtent Tracking

• use “vertex descent”use “vertex descent”

• on convex polytope, local minimizer of extent is also on convex polytope, local minimizer of extent is also    global minimizer   global minimizer

• use “vertex descent”use “vertex descent”

• on convex polytope, local minimizer of extent is also on convex polytope, local minimizer of extent is also    global minimizer   global minimizer

DD

frame t0frame t0

v0v0

frame t1frame t1

v1v1

v0v0

Results: Tumbling ToothpicksResults: Tumbling ToothpicksResults: Tumbling ToothpicksResults: Tumbling Toothpicks

• uniform scale (“us”): add more of same sizeuniform scale (“us”): add more of same size

• biases occlusion complexity superlinearlybiases occlusion complexity superlinearly

• uniform density (“ud”): add more of scaled sizeuniform density (“ud”): add more of scaled size

• occlusion complexity increases linearlyocclusion complexity increases linearly

• uniform scale (“us”): add more of same sizeuniform scale (“us”): add more of same size

• biases occlusion complexity superlinearlybiases occlusion complexity superlinearly

• uniform density (“ud”): add more of scaled sizeuniform density (“ud”): add more of scaled size

• occlusion complexity increases linearlyocclusion complexity increases linearly

• eccentric ellipsoids moving in eccentric ellipsoids moving in    cubical volume   cubical volume

• eccentric ellipsoids moving in eccentric ellipsoids moving in    cubical volume   cubical volume

Performance with Increasing VelocityPerformance with Increasing VelocityPerformance with Increasing VelocityPerformance with Increasing Velocity

0

0.005

0.01

0.015

0.02

0.025

0.03

0.035

0 10 20 30 40 50 60 70

velocity scale

cp

u s

ec

on

ds

ud max ud avg

0

0.5

1

1.5

2

2.5

3

3.5

0 500 1000 1500 2000

number of objects

cp

u s

ec

on

ds

us max us avg ud max ud avg

Performance with Increasing ObjectsPerformance with Increasing ObjectsUniform Density & Uniform ScaleUniform Density & Uniform ScalePerformance with Increasing ObjectsPerformance with Increasing ObjectsUniform Density & Uniform ScaleUniform Density & Uniform Scale

Kd-Tree Culling PerformanceKd-Tree Culling PerformanceKd-Tree Culling PerformanceKd-Tree Culling Performance

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

0 500 1000 1500 2000

number of objects

cp

u s

ec

on

ds

k=1 ud k=3 ud

• six aircraft flying through valleysix aircraft flying through valley

• investigates rendering acceleration at various investigates rendering acceleration at various    levels of terrain splitting   levels of terrain splitting

• update rate assumptions:update rate assumptions:• aircraft parts: 20%aircraft parts: 20%• terrain parts: 70%terrain parts: 70%• sky 40%sky 40%

• six aircraft flying through valleysix aircraft flying through valley

• investigates rendering acceleration at various investigates rendering acceleration at various    levels of terrain splitting   levels of terrain splitting

• update rate assumptions:update rate assumptions:• aircraft parts: 20%aircraft parts: 20%• terrain parts: 70%terrain parts: 70%• sky 40%sky 40%

Results: Canyon FlybyResults: Canyon FlybyResults: Canyon FlybyResults: Canyon Flyby

• aggregation penalty: aggregation penalty:   render every frame  render every frame

• aggregation penalty: aggregation penalty:   render every frame  render every frame

Results: Canyon FlybyResults: Canyon FlybyResults: Canyon FlybyResults: Canyon Flyby

split cpu avg cpu max exp. factor parts agg. update rate

20x20 17.33 29.82 5.04 0.2% 41.8%14x14 8.08 14.59 3.78 1.0% 38.0%10x10 5.17 9.88 3.09 6.4% 40.4%7x7 4.51 9.68 2.58 22.5% 42.4%5x5 5.37 11.01 2.28 53.1% 72.0%

split cpu avg cpu max exp. factor parts agg. update rate

20x20 17.33 29.82 5.04 0.2% 41.8%14x14 8.08 14.59 3.78 1.0% 38.0%10x10 5.17 9.88 3.09 6.4% 40.4%7x7 4.51 9.68 2.58 22.5% 42.4%5x5 5.37 11.01 2.28 53.1% 72.0%

• cpu times are in ms per framecpu times are in ms per frame

• update rate is poly-weightedupdate rate is poly-weighted

• cpu times are in ms per framecpu times are in ms per frame

• update rate is poly-weightedupdate rate is poly-weighted

Newell, Newell, & Sancha (1972)Newell, Newell, & Sancha (1972)Newell, Newell, & Sancha (1972)Newell, Newell, & Sancha (1972)• traverse depth-sorted list of polygonstraverse depth-sorted list of polygons• if next polygon doesn’t overlap, send to outputif next polygon doesn’t overlap, send to output• otherwise use tests of increasing complexity:otherwise use tests of increasing complexity:• screen bounding box, vertex/plane, 2d intersectionscreen bounding box, vertex/plane, 2d intersection

• if unoccluded, polygon is sent to outputif unoccluded, polygon is sent to output• if occluded, polygon is marked and reinsertedif occluded, polygon is marked and reinserted• marked polygons are split to remove cyclemarked polygons are split to remove cycle

• traverse depth-sorted list of polygonstraverse depth-sorted list of polygons• if next polygon doesn’t overlap, send to outputif next polygon doesn’t overlap, send to output• otherwise use tests of increasing complexity:otherwise use tests of increasing complexity:• screen bounding box, vertex/plane, 2d intersectionscreen bounding box, vertex/plane, 2d intersection

• if unoccluded, polygon is sent to outputif unoccluded, polygon is sent to output• if occluded, polygon is marked and reinsertedif occluded, polygon is marked and reinserted• marked polygons are split to remove cyclemarked polygons are split to remove cycle

EE

AA

ZZ

CC

BBFF

HH

GG