04/19/23 Eugene Borovikov, Alan Sussman and Larry Davis, UMCP

A High-performanceMulti-perspective Vision Studio

An Efficient System for Multi-Perspective Imaging and 3D Shape

Analysis

04/19/23 Eugene Borovikov, Alan Sussman and Larry Davis, UMCP

Multi-view vision

• interesting

• affordable

• challenging

• distributed

04/19/23 Eugene Borovikov, Alan Sussman and Larry Davis, UMCP

Multi-perspective environments

Keck Lab• 64 cameras• 85 frames/sec• 1 min = 95GB

04/19/23 Eugene Borovikov, Alan Sussman and Larry Davis, UMCP

Volume reconstruction

• multi-perspective• silhouette-based• visual cone intersection• special octree encoding

04/19/23 Eugene Borovikov, Alan Sussman and Larry Davis, UMCP

Background subtraction

Volume reconstruction

04/19/23 Eugene Borovikov, Alan Sussman and Larry Davis, UMCP

Multi-perspective silhouette extraction

- =

- =

- =

Volume reconstruction

04/19/23 Eugene Borovikov, Alan Sussman and Larry Davis, UMCP

Volume reconstruction

04/19/23 Eugene Borovikov, Alan Sussman and Larry Davis, UMCP

Visual cone construction

image plane

Volume reconstruction

04/19/23 Eugene Borovikov, Alan Sussman and Larry Davis, UMCP

3D occupancy map as octree

image plane

Volume reconstruction

04/19/23 Eugene Borovikov, Alan Sussman and Larry Davis, UMCP

resolution=8depth

Volume reconstruction

04/19/23 Eugene Borovikov, Alan Sussman and Larry Davis, UMCP

Multi-perspective Vision StudioData Capture

Loader

Database

Front-end services

Back-end services

Client Client

Data CaptureFeatures• abstraction from data acquisition• multi-view sequence management• extensible application framework• based on ADR and DataCutter

Applications• Volumetric shape reconstruction• 3D density-based model fitting• Texture mapping surface meshes

04/19/23 Eugene Borovikov, Alan Sussman and Larry Davis, UMCP

Data de-clustering based on Hilbert space-filling curve

camera indextim

e in

dex

1 2 3 4 5 6 7 8

1

2

3

4

5

6

7

8

Customizable Studio Server• Data elements (chunks): image<cam-ndx,time-ndx>• Loader: Hilbert curve based de-clustering algorithm• Parallel back-end: database engine

– index: (x,y,z,t) -> (cam,time)

– aggregation: associative&commutative

• Application front-end: gateway– query: application dependent

– result: AppFE node is optional

04/19/23 Eugene Borovikov, Alan Sussman and Larry Davis, UMCP

Client GUI

04/19/23 Eugene Borovikov, Alan Sussman and Larry Davis, UMCP

Approximate projection Vertex cache Naive projeciton

0

2

4

6

8

10

12

projection methods

seconds

Projection timing statistics, depth 8

Server performance

04/19/23 Eugene Borovikov, Alan Sussman and Larry Davis, UMCP

2

4

8

16

2

4

8

16

0

2000

4000

6000

8000

10000

12000

frame group size

Constant work load performance

number of processors

seco

nd

s

Server performance

04/19/23 Eugene Borovikov, Alan Sussman and Larry Davis, UMCP

A density fitting example

2

22

1

21

2211 ,,,; b

ax

b

ax

eebabaxf

04/19/23 Eugene Borovikov, Alan Sussman and Larry Davis, UMCP

Density based shape modeling

)1,0[: mn RRf

nR

x

x

MdVxf

Dxf

;

;max (consistency)

(conservation)

V

xR

dVxfFFm

; where,maxarg

given a volume V, fit a density f by solving

04/19/23 Eugene Borovikov, Alan Sussman and Larry Davis, UMCP

Hierarchical fitting

04/19/23 Eugene Borovikov, Alan Sussman and Larry Davis, UMCP

Density based modeling results

04/19/23 Eugene Borovikov, Alan Sussman and Larry Davis, UMCP

Mesh texture coloring

04/19/23 Eugene Borovikov, Alan Sussman and Larry Davis, UMCP

ConclusionsMulti-perspective vision studio• abstracting vision application from sensor array• portability across parallel platforms• robustness in handling large datasets• expandable functionalityHigh performance comes from• effective data de-clustering (Hilbert curve)• frame grouping to improve workload balance• efficient voxel projection strategy