Parameter-free/Pareto-driven Procedural 3D Reconstruction

000001002003004005006007008009010011012013014015016017018019020021022023024025026027028029030031032033034035036037038039040041042043044045046047048049050051052053

054055056057058059060061062063064065066067068069070071072073074075076077078079080081082083084085086087088089090091092093094095096097098099100101102103104105106107

CVPR#306

CVPR#306

CVPR 2012 Submission #306. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.

Parameter-free/Pareto-driven Procedural 3D Reconstruction of Buildings fromGround-Level Sequences

Anonymous CVPR submission

Paper ID 306

Abstract

In this paper we address multi-view reconstruction of ur-ban environments using 3D shape grammars. Our formula-tion expresses the solution to the problem as a shape gram-mar parse tree where both the tree and the correspondingderivation parameters are unknown. Besides the grammarconstraint, the solution is guided by an image support thatis twofold. First, we seek for a derivation that induces op-timal semantic partitions in the different views. Second, us-ing structure-from-motion, noisy depth maps can be deter-mined towards minimizing their distance from to the onespredicted by any potential solution. We show how the un-derlying data structure can be efficiently optimized usingevolutionary algorithms with automatic parameter selec-tion. To the best of our knowledge, it is the first time thatthe multi-view 3D procedural modeling problem is tackled.Promising results demonstrate the potentials of the methodtowards producing a compact representation of urban envi-ronments.

1. Introduction

Urban environment modeling has been a burning issue inthe computer vision, remote sensing and geoscience com-munities over the past few decades. While, the theory be-hind stereo and structure-from-motion has became moremature, new research perspectives have emerged. One canthink of the sensational impact of [11] and the many studiesthat followed on browsing large collections of images. An-other challenge that arose recently is related to the recon-struction of large outdoor scenes. To treat efficiently sucha problem, there is need to exploit compact representations.Towards this end, [4] have proposed an hybrid representa-tion, combining predefined primitives (e.g. cylinders) andclassical meshes. However, existing approaches lack of se-mantics while at the same time scalability might be an issuein terms of representation, transmission, etc.

Shape grammars are not so recent, they were intro-

Figure 1. Based one 2D classification data (upper left) and auto-matically computed 3D points cloud (second row), we retrieve thestructural information of the building (right and second row).

duced in a mathematical formalism by [12] as a generativespecification of shape classes in order to analyze paintingsand sculptures. During the two decades that followed, manystudies made use of this framework to decode the unifyingrules behind certain architectural styles [13].

This concept was emancipated in computer graphics, un-der the name of procedural modeling [18, 7]. In this do-main, they were used as random generators of realistic 3Drepresentations of buildings from a given architectural style.

Approximately in the same period, image parsing basedon shape grammars have drawn a great interest. For in-stance, [8] and [3] focused on detecting repetitive structuresto produce a grammatical explanation of an input image.Again in the computer vision domain, [1] and [16] tackledimage parsing as a classification problem where the gram-mar can generate an extensive set (referred to as the proce-dural space) of semantic image segmentations. Last, [15]took up the same philosophy but endeavored to improvethe procedural space exploration relying on reinforcementlearning.

Despite the great interest of these methods, they fail to

1

108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161

162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215

CVPR#306

CVPR#306


provide the real 3D structure. Orthogonally, [17] used shapegrammars to retrieve complex mass models of modern ar-chitectures. Ancient-heritage inverse procedural modelinghas been presented in [6]. In this paper, we focus on au-tomatic multi-view procedural reconstruction of urban 3Dmodels embedding a mass model and faithful facade de-tails. To the best of our knowledge, comparable approachesrely on heavy user interaction [9].

Our approach differs from prior work using shape gram-mars for building reconstruction in terms of output as wellas in terms of means to achieve it. It assumes as input asequence of N images I1, . . . , IN along with their calibra-tion matrices π1, . . . , πN . Similarly to existing approaches,whenN = 1 (a case mainly considered for comparison pur-pose), only 2D parsing can be solved. Otherwise, we can re-cover the 3D structure of the building. To this end, procedu-ral techniques are developed to generate 2D/3D geometrico-semantic layouts to be evaluated with respect to two crite-ria1, leading to a multi-objective optimization where seman-tics and geometry are placed on an equal footing. The firstcriterion, evaluates the likelihood of the visual properties ofthe different architectural elements. The concurrent crite-rion accounts for the consistency between noisy referencedepth-maps (derived through structure-from-motion) andthe ones predicted via the procedural model. The unknownsare expressed as a grammar parse tree and are simultane-ously recovered using evolutionary computation methods.

The remainder of the paper is organized as follows. InSection 2, the grammar formalism is explained and illus-trated on both facade and building procedural modeling.Then, in Section 3, we present our search strategy basedon evolutionary algorithms. Section 4 describes the energyfunctions used to implement the single-view and multi-viewcriteria. Last, we validate our approach in Section 5, wherewe show that it is competitive with the state-of-the-art forthe procedural facade segmentation, and present promisingquantitative/qualitative results for the newly introduced 3Dprocedural reconstruction problem.

2. Procedural Building ModelingThe central idea developed here is that a shape grammar

might be thought as a formal specification of how to ran-domly generate 2D/3D layouts (Figure 2) following somedesign principles. We shall see that such layouts not onlybear a geometric description but also a semantic one, mak-ing them compliant to image understanding techniques.

2.1. A Shape Grammar Paradigm

In this framework, we call shape a collection of prim-itives s = {p1, . . . , pk}. A primitive p is characterizedby a class cp (e.g. floor, window, balcony) and a geomet-

1Note that only one criterion can be evaluated if N = 1.

(a) (b)

Figure 2. Grammars as semantic layout generators: random lay-outs obtained with a 2D grammar (a) and a 3D grammar (b).

ric description (a rectangle in 2D or a mesh in 3D). A ruler : cr → rhs[x](.) describes how to replace a primitivep by primitives produced by its right-hand-side procedurer[x](p), parametrized by a real-valued vector x ∈ Rn.These parameters usually stand for geometric properties,e.g. dimensions, orientations, etc.

Note that r can be applied to p only if cp matches cr. Wedenote Rp = {r s.t. cr = cp} the set of such rules. Basedon this, a primitive (or by extension its class cp) will be saidterminal ifRp is void and non-terminal otherwise.

Then, a shape grammar is defined by an axiom primitive,a set of primitives V called vocabulary and a set of rulesR.Let us now describe how a shape grammar can be turnedinto a stochastic shape factory.

2.2. Derivation of a Shape Grammar

Algorithm 1 Shape grammar derivation processs← {axiom}while ∃p ∈ s s.t.Rp 6= ∅ do

Pick r ∈ Rp randomly (r : cp → rhs[](.))Sample x randomly{p1, . . . , pk} = rhs[x](p)s← s ∪ki=1 pi \ p

end while

The process explained in algorithm 1 describes the ran-dom shape generation. The initial shape is composed of theaxiom only. At each iteration, a non-terminal primitive p isselected in the current shape. A rule r ∈ Rp is randomlychosen along with the values of the necessary parametersx. Then, p is replaced by the primitives emerging from theapplication of r to p. The process stops as soon as the shapeis composed of terminal primitives only. This iterative re-placement process can naturally be represented with a parsetree (Figure 3) where internal nodes are non-terminals, andleaves are either terminals or unprocessed non-terminals.For each leave under processing, the chosen rule applica-tion yields as many children as the produced primitives.

2

216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269

270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323

CVPR#306

CVPR#306


Axiom

Fls1

Fls2

(a) (b)

Figure 3. Parse tree (a) associated with a partial derivation (b) ofthe grammar in Table 1. Terminals correspond to colored nodes.

Implementing the derivation process provides a randomgenerator of shapes following the formal specification en-coded in the grammar. The resulting shape depends directlyon the random choices made during the derivation. In theperspective of optimizing them, we store the rule applied toany node and its parameters x in the concerned node.

2.3. 2D Modeling of Facades

Following the ideas developed in [16], we can use thisprocedural framework to decompose a frontal-facade imageinto a set of labeled rectangles. The labels corresponds tothe classes of the primitives, and their geometry is alwaysrectangular. Here we provide a simple 2D grammar to helpthe reader’s understanding of the procedural paradigm andgive concrete examples of procedures used in the rules.

The grammar rules are provided in Table 1 and few stepsof a derivation are shown in Figure 4. The only proceduresused here are the x/ySplit that carve a primitive into twoalong the x/y direction (e.g. left image in Figure 4).

r1 Axiom → ySplit[hc] Fls1,Rfr2 Fls1 → ySplit[h] Wa,Fls2

r3 Fls2 → ySplit[h’] Fl1,Fls1

r4 Fl1 → xSplit[w] Wa,Fl2

r5 Fl2 → xSplit[w’] Wi,Fl1

Table 1. A toy example of 2D grammar.

A more complex 2D grammar was actually designed toexpress sophisticated facade layouts (see Figure 2 (a)). Inthis version, additional procedures create dynamic links be-tween the parameters of the rules at different scopes in thederivation so as to enforce symmetries and alignments. Thiskind of dynamic constraints generalizes the grammar factor-ization concept proposed in [16] and turns out to be moreflexible towards 3D modeling of buildings.

2.4. 3D Modeling of Buildings

To generate 3D layouts as depicted in Figure 2 (b), wefollow the general scheme proposed by [7]: build a mass

Figure 4. Few steps of a 2D grammar derivation.

model from a footprint, and then refine this coarse represen-tation by further splitting each facade in both directions tofirst grow the floors, then the windows, wall and balconiesand other terminal elements of the facade.

For conciseness, the rules of the grammar are not listedhere2. They rely mainly on 7 kinds of procedures: Moveand Scale change the position and scale of a primitives,Extrude and Facetize switch between 2D and 3Dprimitive, Insert plugs a new primitive in replacement ofanother, Roof creates a mansard roof over a polygon andx/ySplit.

Once the grammar has been defined, the next step con-sists in searching among the possible designs the most suit-able one to account for the images.

3. Efficient Exploration Strategy through Evo-lutionary Algorithms

In this section, we introduce an elegant and appropriateinference framework based on evolutionary algorithms. Re-cent surveys of this field can be found in [5].

3.1. Evolutionary Algorithms

Evolutionary algorithms are part of meta-heuristicswhich are stochastic search optimization methods. A funda-mental notion in this strategy refers to elitism which favorspromising individuals. The underlying concept of these al-gorithms is presented in Figure 5. Each iteration aims atimproving a generation P made of previously evaluated in-dividuals. Evaluation of an individual consists in computingits objective function value(s). Then promising individualsP ∗ are selected for mating (recombination) leading to O asoffspring. After a subsequent mutation step, the offspringO is evaluated, and reinserted in the current population to-wards replacing (partly) the previous generation.

3.2. Optimization of the Grammar Derivation

To apply the previous methodology to our particularproblem, we need to specify the kind of individuals we arestudying and the evolution processes applied to them.

2This grammar and the 2D one are detailed in supplementary material.

3

324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377

378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431

CVPR#306

CVPR#306


Figure 5. Typical pipeline of evolutionary algorithms.

One usually discriminates between two representations.On the one hand, the genotype is convenient to operate evo-lutions. On the other hand, the phenotype is a “physical”representation, more natural for fitness evaluation.

Genotype

A parse tree is made of nodes involving three componentsn = (r,x,p). r and x are the variables to be optimized. Thelast component, p, is the semantic primitive created dur-ing the derivation and is entirely determined by the nodesabove n in the tree. For a given node, the rule can be cho-sen among the finite setRp of rules applicable to p. On thecontrary, the parameters correspond to a real-valued vector.

A mutation acts on a single individual. After a noden = (r,x, p) is chosen randomly, two options are available.

1. Rule mutations consist in replacing r by a random ruleofRp, if #Rp > 1.

2. Parameter mutations add a uniform perturbation u to arandomly chosen element xi ∈ x:

xi ← xi + u, (1)

To recombine two individuals, we select two nodes nand n′ (one per tree), such that cp = cp′ . Then their contentsr, x and r′, x′ are swapped.

Note that mutations and recombinations can jeopardizethe validity of specific sub-trees. Any invalid sub-tree ispruned and re-derived at random.

Phenotype and evaluation

A parse tree stores all the necessary information to derivea semantic layout L (Figure 2). This layout is the physical

expression of the individual: its phenotype. In the single-view case, the individual relevance with respect to the ob-servations will be evaluated thanks to an appearance energyEa(L). In the multiple-view case, an individual will be alsoevaluated thanks to a depth energy Ed(L). The energy def-initions will be detailed later.

Based on the fitness of any individual is known, an elitistprocess called selection is designed to ensure the preserva-tion of strong features. When only Ea(L) is considered, in-dividuals are selected by running 2-tournaments. Two can-didates are randomly picked and the fittest is kept as a gen-itor (i.e. appended to P ∗).

Otherwise, Etotal(L) = (Ea(L), Ed(L)) ∈ R2 we cancompare individuals using Pareto partial ordering. For-mally, we will note L1 >p L2, if L1 has all energies smallerthanL2. Given this partial ordering, we seek to approximatea set called the Pareto frontier. It consists of all individualsthat are dominated by none other solution (Figure 6).

Among different options, we have chosen the modifiedStrength Pareto Evolutionary Algorithm (SPEA-II) selec-tion scheme [19]. The process (Algorithm 2) performs boththe pareto front estimation and the selection of genitors.Selection is performed by using a classical tournament onthe current estimate of the front. The front is approximatedby a fixed size archive A, which is initialized with all non-dominated solutions found so far and it then pruned or com-pleted at need. Note that diversity is promoted among gen-itors by preferably pruning solutions located at dense posi-tions in the current Pareto front estimate.

Algorithm 2 SPEA-II algorithmRequire: P : current populationRequire: A: archive of non-dominated solutionsRequire: a: maximum size of archiveA← non-dominated solutions of P ∪Aif #A < a then

extend A with (a − #A) fittest dominated solutionsfrom P

elseremove from A its (#A − a) individuals at dense lo-cations

end ifP ∗ ← tournament selection on Areturn P ∗, A

3.3. Optimal solution in the multi-objective case

Successive iterations of the multi-objective algorithmproduce a set of candidates approximating the Pareto fron-tier. In practice, we are looking for a unique optimal solu-tion. Thus, we combine the two energies into a single one:

E(L) = αEa(L) + (1− α)Ed(L) (2)

4

432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485

486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539

CVPR#306

CVPR#306


Ed

Ea

Admissible solutions

Figure 6. Illustrative example: Pareto frontier (red line) and its es-timate (red triangles). Level sets of a linearized fitness E (dashedblack). A local minimum of E (purple cross) far away from theglobal one (orange cross).

The level sets of E correspond to dashed straight linesin Figure 6. The optimal solution with respect to this en-ergy is easily determined from the complete set of Paretosolutions. Note that among the approximate Pareto solu-tions, only those belonging to the Pareto convex hull mightbe optimal for a linear combination of the objectives.

Better exploration behavior

One may think that we could have used a combined energyfrom the beginning and rely on classical tournaments. Thisassumption is strongly objected in the theory of evolution-ary optimization where the consensus states that keepingmultiple objectives reduces the risk of early convergence toa local minimum. Concerning our problem, the validity ofthis statement has been observed experimentally (Figure 7).A typical case of local minima is illustrated in Figure 6where the Pareto set is concave.

Automatic weight selection

It is also important to note that the choice of α plays akey role on the final outcome of the optimization process.Defining such a value beforehand is not straightforward.The knowledge of the Pareto set makes this task easier sinceits bounding box brings valuable insights regarding the ap-propriate balance between the two energy components. Apractical value of α corresponds to iso-lines of E that areparallel to the diagonal of the Pareto set bounding box.

4. Concurrent Energy ModelsIn this section, we propose two energy models, one ex-

tending the appearance model proposed in [15] to non-frontal views of a 2D/3D layout and another evaluating thequality of a 3D layout with respect to depth profiles.

Ed

Ea

Figure 7. A population driven by SPEA-II in orange and onedriven by a combined energy in cyan. Red/blue dots correspondthe last generations and red line to the estimated Pareto set.

4.1. Appearance Energy

The first energy makes the link between the symbolicworld of grammars and the statistical visual properties ofthe associated architectural elements. It can be used as soonas a calibrated image of the building is available. Therefore,it is adapted to facade segmentation and 3D reconstruction.

We adopt a Bayesian formulation where we aim at defin-ing the posterior probability of the procedural layout Lknowing the set of views I: P (L|I). We first express theposterior with the likelihood and the prior using the Bayesrule: P (L|I) ∝ P (I|L)P (L). Given that the grammar al-ready expresses a very strong prior on the semantic layout,we do not consider additional ones. Then, assuming inde-pendence between the different views and among the pixelsof each image, the likelihood is factorized as:

P (I|L) =

N∏k=1

P (Ik|L) =

N∏k=1

∏x

P (Ik(x)|L) (3)

In the factorization, we only have to express the pixellikelihood P (Ik(x)|L). We make the natural assumptionthat the distribution of Ik(x) given the model L only de-pends on the voxel π−1k (x) which projects on the pixel x inIk. Then P (Ik(x)|L) can be rewritten as P (Ik(x)|π−1k (x)).

In the single-view case, π1 is a one-to-one mapping fromthe image domain to the 2D layout. Otherwise the πk’s arenot necessarily bijective and the latter expression is onlyvalid when π−1k (x) exists (i.e. when the line of sight passingthrough x intersects L). In the opposite case, π−1k (x) isconsidered latent and P (Ik(x)|L) is averaged over the setof admissible values for π−1k (x).P (Ik(x)|π−1k (x)) is estimated based on the semantics

predicted by L at x i.e. as P (Ik(x)|c(π−1k (x))). In our set-tings, for any class c the probability distribution P (Ik(x)|c)is learned using a Gaussian Mixture Model on the RGB val-ues of pixels in a training set. In practice, the training set is

5

540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593

594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647

CVPR#306

CVPR#306


created by a user who paints some brush strokes on one ofthe input images for all the terminal classes of the grammar.

Last, the appearance energy Ea is obtained by taking thenegative log-likelihood as:

Ea(L) =∑k

∑x

− log (P (Ik(x)|L)) (4)

It uses appearance as a way to distinguish between theterminal semantic elements. However, it fails to accountfor real 3D evidence with respect to the voxels belonging todifferent or even to the same class when located at differentdepths. This can be addressed through a depth energy.

4.2. Depth Energy

As an alternative, a depth energy is obtained by compar-ing the model L with the reference 3D point cloud Pref .For each visible facade, we extract two depth maps. Oneis derived from the point cloud while the second corre-sponds to L. Practically, each depth map is extracted fromthe Z-buffer of a virtual orthographic camera facing thefacade. We end up with a list of pairs of depth maps(D1L, D

1ref ), . . . , (DM

L , DMref ), where M is the number of

facades. Then the energy of L is:

Ed(L) =∑m

∑x

||DmL (x)−Dm

ref (x)− dm||2, (5)

where dm is the mean distance between DmL and Dm

ref .Therefore, the energy is minimal when both depth mapsdiffer by a constant (not required to be null for better ro-bustness to small inaccuracies in the calibration data).

5. Experimental Validation5.1. Single-view Performance

The following tests were launched on the data-set pro-posed by [16]. 10000 hypotheses were tested for each fa-cade. Motivated by empirical observations, the populationand offspring sizes were set respectively to 32 and 16.

We computed the confusion matrix, where at row i andcolumn j is the percentage of pixels attached to class i ac-cording to the ground truth and to j with our approach.

70 23 5 0 2 0 02 91 6 0 0 0 17 19 73 0 0 0 10 1 0 76 0 0 226 4 0 0 88 2 01 0 0 0 3 96 00 2 0 4 0 0 93

[16] [15] cp−11 −11 window+8 +7 wall+1 +10 balcony+5 −8 door+8 +2 roof+2 +2 sky−1 −4 shop

Comparisons with [16, 15] regarding detection rates (di-agonal entries) are shown in blue/red. This experiment

shows that evolutionary algorithms provide equivalent per-formance as both alternatives, with a competitive number ofhypotheses as in the state-of-the-art. This is very promisingfor the 3D multi-view extension.

5.2. Multi-view Quantitative Validation

Settings

We use tens of images to run the automatic structure-and-motion tool Bundler [10]. Then, a dense point cloud is gen-erated using the PMVS tool [2]. For practical reasons, thesesteps can be performed for multiple buildings at once. Af-terward, to reduce the computational burden, we limit our-selves to 2 views per visible facade.

The grammar axiom is derived from a footprint retrievedfrom OpenStreetMap, using the address of the targetedbuilding. The footprint must be expressed in the same eu-clidean coordinate system as the calibrated cameras. Wehave met this requirement by providing few correspon-dences between the cadastrial map and the structure-from-motion point cloud. Note that this step could be automated[14] and allows the output building models to be seamlesslyembedded in a complete GIS environment.

Our complete data set is made of 10 buildings. For agiven building, a ground-truth procedural layout Lgt can bebuilt manually. This task is time-consuming and was donecarefully for each building of our data set.

The evolutionary algorithm runs 200 generations pro-ducing each at most 100 new individuals.

Performance criteria

Given that the model incorporates geometric and semanticinformation, it is natural to evaluate the relevance of bothaspects. Regarding the semantic aspect, we keep the sameline with confusion matrices, although the ground-truth la-beling is derived from Lgt.

To assess geometric accuracy, we can compute the av-erage point-to-surface distance between the inferred modeland the reference one.

d(L,Lgt) =1

#L∑x∈L

minxgt∈Lgt

d(x, xgt). (6)

Results

We first consider the classification criterion presented in thefollowing confusion matrix. Globally, the numerical valuesare equivalent to these obtained in the single-view experi-ments. We do not provide a detailed comparison with theprevious experiments as the data sets differ.

6

648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701

702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755

CVPR#306

CVPR#306


70 24 5 0 1 03 83 13 0 0 010 7 82 0 1 00 2 0 84 0 148 6 7 0 79 00 4 0 2 0 94

windowwallbalconydoorroofshop

In Table 2, we show the statistics obtained with Equa-tion 6 on a per-semantics basis. Overall, the deviation isbounded by 30 centimeters. Large gaps between the groundtruth and the inferred model concern mainly the roof and theshops. But in such cases, this difference has little impact onthe visual quality of the model since the corresponding ar-chitectural elements are large. On the contrary, the errorfor windows averages to 11 centimeters and is less satisfac-tory given that the depth of a window amounts to approxi-mately 50 centimeters. Nevertheless when considering thefinal model, this is hardly noticeable.

c dc(L,Lgt)window 11cmwall 4cmbalcony 13cmdoor 1cmroof 31cmshop 27cm

Table 2. Geometric accuracy.

We have allowed 20, 000 hypotheses which shows thatdespite a greater complexity in the inference task, the num-ber of candidates remains comparable to state-of-the-art fa-cade parsing [15]. However, even though the algorithmiccomplexity is harnessed, the necessary time to reconstructa single building remains significant mainly because of thecomputation of the energy terms. The inference typicallyrequires around one hour.

5.3. Examples of Multi-view Reconstructions

We present here a few qualitative results on differentbuildings from Paris. For each building, we show the clas-sification induced by the 3D semantic layout and the tex-tured 3D model in which structural elements are replacedby highly detailed models. In particular, Figure 8 draws aparallel between these results and the raw classification anddepth cues. The gain is obvious with respect to both as-pects. Despite the complexity of the problem and the verynoisy level of appearance and depth information, the mod-els are very accurate in terms of geometry and of topology.Different results obtained on other examples are depicted inFigure 9. Last, Figure 10 shows the reconstruction obtainedwith a building presenting many street facades. This exam-ple illustrates the benefit of handling the building as a wholeinstead of dealing with each facade separately.

(a) raw classification

(b) Optimal 3D layout

(c) raw 3D

(d) Optimal 3D model

Figure 8. Optimal procedural reconstruction compared to raw in-puts. First row: raw classification and point cloud. Second row:the optimal 3D layout and the derived 3D model obtained byadding detailed 3D models and back-projecting a texture.

Although the results are very satisfying, few failure casesmust be deplored. They occur when the actual layout can-not be generated by the grammar, as for instance balconiesrunning only partially along the facade (third floor in Fig-ure 8 (b)). In such a case, the reconstruction is a consensusbetween the language of the grammar and the observations.Besides, the registration of the point cloud and the camera issometimes not as accurate as expected, and it can thereforeintroduce some bias in the final model.

6. ConclusionIn this paper, we have introduced an innovative

grammar-based approach to multi-view 3D reconstruction.Our method inherits the strength of grammar representa-tions like the modularity with respect to the class of ar-chitectures that could be derived, the compactness of therepresentation and the semantic understanding of the en-vironment. This is achieved through an aggregation ofsegmentation and reconstruction objectives. The resultingparadigm opposite to prior grammar-based segmentationmethods provide the real 3D geometry of the scene.

As a by-product, we have demonstrated the potentials ofevolutionary algorithms to infer optimal grammar deriva-tion. Because of its flexibility, this paradigm is amenable tomany extensions. For instance, the estimation of the cameraparameters on the fly, as well as the footprint of the buildingas part of the evolutionary algorithm could lead to a fullyautomatic approach to 3D modeling. We are also consider-ing the fusion of range data with aerial/street-level images

7

756757758759760761762763764765766767768769770771772773774775776777778779780781782783784785786787788789790791792793794795796797798799800801802803804805806807808809

810811812813814815816817818819820821822823824825826827828829830831832833834835836837838839840841842843844845846847848849850851852853854855856857858859860861862863

CVPR#306

CVPR#306


Figure 9. 3D reconstruction of different buildings.

as another extension. Eventually, looking at efficient meansof accelerating the convergence process through data-drivenmutations is a very promising direction.

References[1] F. Alegre and F. Dellaert. A probabilistic approach to the

semantic interpretation of building facades. In InternationalWorkshop on Vision Techniques Applied to the Rehabilitationof City Centres, 2004. 1

Figure 10. 3D reconstruction of a building with multiple facadesvisible from the street. Note that floors align seamlessly along thedifferent facades.

[2] Y. Furukawa and J. Ponce. Patch-based multi-view stereosoftware. http://grail.cs.washington.edu/software/pmvs/. 6

[3] P. Koutsourakis, L. Simon, O. Teboul, G. Tziritas, andN. Paragios. Single view reconstruction using shape gram-mars for urban environments. In ICCV, 2009. 1

[4] F. Lafarge, R. Keriven, M. Bredif, and V. Hiep. Hybrid multi-view reconstruction by jump-diffusion. In CVPR, 2010. 1

[5] S. Luke. Essentials of Metaheuristics. Lulu, 2009. 3[6] M. Mathias, A. Martinovic, J. Weissenberg, and L. Gool.

Procedural 3d building reconstruction using shape grammarsand detectors. In 3DIMPVT, 2011. 2

[7] P. Muller, P. Wonka, S. Haegler, A. Ulmer, and L. Van Gool.Procedural modeling of buildings. ACMTOG, 2006. 1, 3

[8] P. Muller, G. Zeng, P. Wonka, and L. Van Gool. Image-basedprocedural modeling of facades. ACMTOG, 2007. 1

[9] F. Niccolucci, M. Dellepiane, S. P. Serna, H. Rushmeierand,and L. V. Gool. Reconstructing and exploring massive de-tailed cityscapes. In VAST, 2011. 2

[10] N. Snavely. Structure from motion for unordered image col-lections. http://phototour.cs.washington.edu/bundler/. 6

[11] N. Snavely, S. M. Seitz, and R. Szeliski. Modeling the worldfrom internet photo collections. IJCV, 2007. 1

[12] G. Stiny and J. Gips. Shape grammars and the generativespecification of painting and sculpture. Information process-ing, 1972. 1

[13] G. Stiny and W. Mitchell. The palladian grammar. Environ-ment and Planning B, 1978. 1

[14] C. Strecha and P. Fua. Dynamic and Scalable Large ScaleImage Reconstruction. In CVPR, 2010. 6

[15] O. Teboul, I. Kokkinos, P. Koutsourakis, L. Simon, andN. Paragios. Shape grammar parsing via reinforcementlearning. In CVPR, 2011. 1, 5, 6, 7

[16] O. Teboul, L. Simon, P. Koutsourakis, and N. Paragios. Seg-mentation of building facades using procedural shape priors.In CVPR, 2010. 1, 3, 6

[17] C. Vanegas, D. Aliaga, and B. Benes. Building Reconstruc-tion using Manhattan-World Grammars. In CVPR, 2010. 2

[18] P. Wonka, M. Wimmer, F. Sillion, and W. Ribarsky. Instantarchitecture. ACMTOG, 2003. 1

[19] E. Zitzler, M. Laumanns, L. Thiele, and Others. SPEA2: Im-proving the strength Pareto evolutionary algorithm. In Euro-gen, 2001. 4

8

Documents

Parameter-free/Pareto-driven Procedural 3D Reconstruction