Semantizing Complex 3D Scenes using Constrained Attribute...

Semantizing Complex 3D Scenesusing Constrained Attribute Grammars

A. Boulch, S. Houllier,R. Marlet and O. Tournaire

slight variant of the materialpresented at SGP 2013

Motivation

Semantizing complex objects in 3D scenes

Bare geometry

Semantized geometry

2

Motivation

Semantizing complex objects in 3D scenes

Bare geometry

Semantized geometry

(rendered)

2

Motivation

Semantizing complex objects in 3D scenes

⌅ Building industry⇤

for the renovation market

⌅ point cloud ! building model

⇤for architects

⌅ building sketch ! rendering⌅ Game industry

⇤for graphic designers

⌅ basic level design ! rendering⌅ Object mining in shape databases

⇤for semantic queries

⌅ 3D object ! semantic labeling

Semantized geometry

2

Motivation

Semantizing complex objects in 3D scenes

⌅ Building industry⇤

for the renovation market

⌅ point cloud ! building model⇤

for architects

⌅ building sketch ! rendering

⌅ Game industry⇤

for graphic designers

⌅ basic level design ! rendering⌅ Object mining in shape databases

⇤for semantic queries

⌅ 3D object ! semantic labeling

Semantized geometry

(rendered)

2

Motivation

Semantizing complex objects in 3D scenes

⌅ Building industry⇤

for the renovation market

⌅ point cloud ! building model⇤

for architects

⌅ building sketch ! rendering⌅ Game industry

⇤for graphic designers

⌅ basic level design ! rendering

⌅ Object mining in shape databases⇤

for semantic queries

⌅ 3D object ! semantic labeling

Semantized geometry

(rendered)

2

Motivation

Semantizing complex objects in 3D scenes

⌅ Building industry⇤

for the renovation market

⌅ point cloud ! building model⇤

for architects

⌅ building sketch ! rendering⌅ Game industry

⇤for graphic designers

⌅ basic level design ! rendering⌅ Object mining in shape databases

⇤for semantic queries

⌅ 3D object ! semantic labelingSemantized geometry

2

Outline

Constrained attribute grammars

Scene interpretation

Bottom-up parsing

Experiments

3

Outline

Constrained attribute grammars

Scene interpretation

Bottom-up parsing

Experiments

4

Grammars to express hierarchical decomposition

Sentence

Subject Verb Complement

NounAdjective

The super computer rules Discovery One

Building

Facade Roof Floor

Window WallArticle

4

Complex, non-hierarchic relations between components

Sentence

Subject Verb Complement

NounAdjective

The super computer rules Discovery One

Building

Facade Roof Floor

Window Wall

agreement (singular)

adjArticle

5

Constrained attribute grammars

G = (N,T ,P , S)

⌅ N: nonterminals ($ complex forms) e.g., window, wall

⌅ T : terminals ($ geometric primitives) e.g., polygon, cylinder

⌅ P : production rules ($ hierarchical decomposition and constraints)⌅ S : start symbols ($ root shapes) e.g., building

6

Basic rules

Decomposition of a complex object y of type Y into its constituentsxi of type Xi :

Y y �! X1

x1

, . . . ,Xn xn

Example:

step s �! riser r , tread t

Rule application:⌅ top-down view:

y decomposes into x1

, . . . , xn⌅ bottom-up view:

given some x1

, . . . , xn, create a new object y

riser

tread

7

Basic rules

Decomposition of a complex object y of type Y into its constituentsxi of type Xi :

Y y �! X1

x1

, . . . ,Xn xn

Example:

step s �! riser r , tread t

Rule application:⌅ top-down view:

y decomposes into x1

, . . . , xn⌅ bottom-up view:

given some x1

, . . . , xn, create a new object y

riser

tread

tread ?

riser ?

7

Constraints

Conditional rule application (conjonction of predicates):

Y y �! X1

x1

,X2

x2

, . . . h cstr1

(x1

), cstr2

(x1

, x2

), . . . i

Example:

riser r �! polygon p h vertical(p) itread t �! polygon p h horizontal(p) istep s �! riser r , tread t

h edgeAdj(r , t), above(t, r) i

riser

tread

8

Attributes

Features attached to each grammar element:⌅ at creation time (primitives)⌅ at rule application (synthetized attributes)

Examples:⌅ length, width⌅ bounding box⌅ ... r.widthr.length

9

Predicates

Predicates on grammar elements:⌅ adj, edgeAdj⌅ orthogonal, parallel⌅ vertical, horizontal⌅ ...Predicates on attributes:⌅ �, >, , ...⌅ =, 6=⌅ ...

t.length

r.length

Example: r .length == t.length

10

Collections of similar elements

Grouping elements via recursion (Y as set of X s):Y y �! X x

Y y �! X x ,Y y2

Grouping elements via specific collection operators:Y y �! coll(X ) xs

Useful operators: maximal collections⌅ maxconn: maximal set of connected components⌅ maxseq: maximal sequence⌅ ...Example:

stairway sw �! maxseq(step, adjEdge) ss

11

Collections of similar elements

Grouping elements via recursion (Y as set of X s):Y y �! X x

Y y �! X x ,Y y2

Grouping elements via specific collection operators:Y y �! coll(X ) xs

Useful operators: maximal collections⌅ maxconn: maximal set of connected components⌅ maxseq: maximal sequence⌅ ... [see complete stairway grammar]

Example:stairway sw �! maxseq(step, adjEdge) ss

11

Outline

Constrained attribute grammars

Scene interpretation

Bottom-up parsing

Experiments

12

Scene interpretation: parse tree

Tree representation of a grammatical analysis of the scene:⌅ leaves: terminals representing primitives⌅ non-leaf nodes: instanciations of grammar rules

12

Scene interpretations: parse forest

Set of parse treeswith systematic sharing (DAG)

Compact representationof all possible interpretations

13

Ambiguity and the exclusivity constraint

Example (assuming no height ordering constraint):

step s �! riser r , tread t h edgeAdj(r , t) i step 2

step 1

Exclusivity constraint:

at most 1 occurrenceof a grammar elementper interpretation

Stairway

Step 2Step 1

Riser 2 Tread 1 Riser 1

14

Outline

Constrained attribute grammars

Scene interpretation

Bottom-up parsing

Experiments

15

Parse forest computation

Bottom-up parsing:construction of the parse forestfrom leaves (terminals) to roots (start symbols)

⌅ create one terminal for each primitives⌅ iteratively create new grammar elements from existing ones

⇤e.g., given grammar rule step s �! riser r , tread t

given existing instances riser r23, tread t18create new instance step s5

⌅ merge identical trees on the fly⌅ stop iterating when no rule applies

15

Rule application order

⌅ Simple rules: use any order⌅ Maximal operators: wait for all subelements to ensure maximality

⇤reverse topological sort of nonterminal dependency graph

maxseq

16

Rule application order

⌅ Simple rules: use any order⌅ Maximal operators: wait for all subelements to ensure maximality

⇤reverse topological sort of nonterminal dependency graph

maxseq

16

Rule application order

⌅ Simple rules: use any order⌅ Maximal operators: wait for all subelements to ensure maximality

⇤reverse topological sort of nonterminal dependency graph

maxseq

16

Mastering the combinatorial explosion

Usual drawback of bottom-up analysis: combinatorial explosion⌅ all trees⌅ all sets, all sequences, ...⌅ all combinations

Our solution:⌅ tree sharing: construction of a parse forest (exp. ! lin.)⌅ maximal operators, with efficient implementation (> exp. ! polyn.)⌅ constraint propagation: predicate ordering ) early pruning

17

Maximal operators

maxseq

vs

allseq

18

Maximal operators

maxseq

vs

max(allseq)

18

Constraint propagation as predicate orderingA simple 2D example

pair p �! seg s1

, seg s2

, h orthogonal(s1

, s2

), adj(s1

, s2

), vertical(s1

) i

19

Constraint propagation as predicate orderingExample of poor ordering

pair p �! seg s1

, seg s2

, h orthogonal(s1

, s2

), adj(s1

, s2

), vertical(s1

) i

1. orthogonality

2. adjacency3. verticality

Complexity:O(#seg2)

constraints to test

20

Constraint propagation as predicate orderingExample of poor ordering

pair p �! seg s1

, seg s2

, h orthogonal(s1

, s2

), adj(s1

, s2

), vertical(s1

) i

1. orthogonality

2. adjacency3. verticality

Complexity:O(#seg2)

satisfied constraints

20

Constraint propagation as predicate orderingExample of poor ordering

pair p �! seg s1

, seg s2

, h orthogonal(s1

, s2

), adj(s1

, s2

), vertical(s1

) i

1. orthogonality2. adjacency

3. verticality

Complexity:O(#seg2 +

#seg ⇥maxDeg)

= O(#seg2)

constraints to test

20

Constraint propagation as predicate orderingExample of poor ordering

pair p �! seg s1

, seg s2

, h orthogonal(s1

, s2

), adj(s1

, s2

), vertical(s1

) i

1. orthogonality2. adjacency

3. verticality

Complexity:O(#seg2 +

#seg ⇥maxDeg)

= O(#seg2)

satisfied constraints

20

Constraint propagation as predicate orderingExample of poor ordering

pair p �! seg s1

, seg s2

, h orthogonal(s1

, s2

), adj(s1

, s2

), vertical(s1

) i

1. orthogonality2. adjacency3. verticality

Complexity:O(#seg2 +

#seg ⇥maxDeg +#seg)

= O(#seg2)constraints to test

20

Constraint propagation as predicate orderingExample of poor ordering

pair p �! seg s1

, seg s2

, h orthogonal(s1

, s2

), adj(s1

, s2

), vertical(s1

) i

1. orthogonality2. adjacency3. verticality

Complexity:O(#seg2 +

#seg ⇥maxDeg +#seg)

= O(#seg2)satisfied constraints

20

Constraint propagation as predicate orderingExample of good ordering

pair p �! seg s1

, seg s2

, h orthogonal(s1

, s2

), adj(s1

, s2

), vertical(s1

) i

1. verticality

2. adjacency3. orthogonality

Complexity :O(#seg)

constraints to test

21

Constraint propagation as predicate orderingExample of good ordering

pair p �! seg s1

, seg s2

, h orthogonal(s1

, s2

), adj(s1

, s2

), vertical(s1

) i

1. verticality

2. adjacency3. orthogonality

Complexity :O(#seg)

satisfied constraints

21

Constraint propagation as predicate orderingExample of good ordering

pair p �! seg s1

, seg s2

, h orthogonal(s1

, s2

), adj(s1

, s2

), vertical(s1

) i

1. verticality2. adjacency

3. orthogonality

Complexity :O(#seg +

#seg ⇥maxDeg)

= O(#seg ⇥maxDeg)

constraints to test

21

Constraint propagation as predicate orderingExample of good ordering

pair p �! seg s1

, seg s2

, h orthogonal(s1

, s2

), adj(s1

, s2

), vertical(s1

) i

1. verticality2. adjacency3. orthogonality

Complexity :O(#seg +

#seg ⇥maxDeg +#seg ⇥maxDeg)

= O(#seg ⇥maxDeg) ⌧ O(#seg2)satisfied constraints

21

Constraint propagation as predicate ordering

1. Unary predicates

2. Invertible predicates that are partially instantiated3. General predicates

= constraints implying only 1 element: complexity O(#element)

Example:riser r �! polygon p h vertical(p) i

22

Constraint propagation as predicate ordering

1. Unary predicates2. Invertible predicates that are partially instantiated

3. General predicates

= constraints with small cardinality when some arguments are fixed

Example:

step s �! riser r , tread t h edgeAdj(r , t) i

22

Constraint propagation as predicate ordering

1. Unary predicates2. Invertible predicates that are partially instantiated3. General predicates

= remaining relations

Example:

step s �! riser r , tread t h orthogonal(r , t) i

22

Outline

Constrained attribute grammars

Scene interpretation

Bottom-up parsing

Experiments

23

Semantization pipeline

Semantizedmodel

Polygonextraction Parsing

Grammar

CAD Model(triangle soup)

Point cloud Primitivedetecion

23

CAD modelsPreprocessing

⌅ Region growing over trianglesfor polygon creation

⌅ Computation of exact andapproximate adjacency graphs

24

CAD modelsDetection of stairs (more examples in supplem. material)

25

CAD modelsDetection of walls, roofs and openings (more examples in supplem. material)

26

Real data: photogrammetry⌅ Preprocessing (point cloud)

⇤clustering using RANSAC

(or region growing)

⇤polygons bounded

by alpha shapes

⌅ Problems:⇤

missing primitives

⇤false primitives

⇤wrong adjacencies

⌅ Solution:⇤

use of a relaxed grammar

⌅ looser bounds⌅ 1-2 missing items OK

⇤22 openings out of 31

27

Quasi-real data: simulated LIDAR

⌅ Planes by region growing in depth image⌅ Polygons as oriented bounding rectangles⌅ Adjacency based on pixels in depth image

28

Size and parsing time (CAD models)

# of # of Parsing time (s)Name triangles polygons stairs openingsLcG 48332 9705 5 15LcA 111979 26585 14 42LcC 385541 111732 33 306LcD 313012 75257 25 111LcF 286996 84347 39 322

29

Precision and recall (%, CAD models)

# of # of Stairs OpeningsName stairs steps Prec. Rec. # Prec. Rec.LcG 3 45 100 93 83 100 90LcA 6 84 100 100 62 98 83LcC 30 210 100 100 196 100 98LcD 5 61 93 100 74 100 93LcF 7 98 100 50 99 100 96

30

Future work

⌅ Principled way to deal with partial or missing primitives⌅ Exploitation of occlusion/visibility information⌅ Scoring of interpretations: pick best tree(s)

31

Conclusion

Constrained attribute grammars:⌅ appropriate to semantize complex objects⌅ high-level specification language

⇤being expert is enough, computer scientist not required

⌅ efficient even on large models

This work:⌅ well-delimited first step: perfect data⌅ extensions required for incomplete/noisy data

On the web⌅ http://imagine.enpc.fr/⌅ sites.google.com/site/boulchalexandre/

32