3D MESH RECONSTRUCTION USING PHOTOGRAMMETRY

Afonso Maria C. F. A. Gonçalves

USING PHOTOGRAMMETRY EX. 1 VISUAL SFM + MESHLAB

Afonso Maria C. F. A. Gonçalves | 20130528

ADVANCED STUDIES PROGRAM IN COMPUTATION APPLIED TO ARCHITECTURE, URBAN PLANNING AND DESIGN

3D SCANNING (2C / 3C)

NOV2013

| 1

_INTRODUCTION

The objective of this exercise was to explore the use of photogrammetry to survey a physical object and use the data collected to generate a point cloud and subsequent 3D mesh reconstruction. For this end, we were asked to choose a small/medium scale object, around the campus premises, as our case study, and then to photograph it from various different angles using a digital camera. For the best 3D reconstruction there needs to be a considerable amount of information and recognized features in all our photos, otherwise the point cloud may not be dense enough or may have considerable gaps in it. Ideally, for a proper reconstruction, one should always avoid using either vey blurry or out-of-focus photos, objects with very reflective or transparent surfaces or moving objects (trees, people, vehicles). The photos taken should also be as consistent as possible, have a good exposure, constant focal length and be properly focused and centered. For this project I decided to choose one of the three carved stone medallions on display in the entrance hall of pavilion 1 (fig. 1). The medallion has relative small amount of detail which, allied to its rough non-specular surface, makes this a good case study. The only drawback was the fact that this medallion was too close to a wall, which made the capture of the back side impossible.

FIG.1 | Carved stone medallion

The method here described has essentially two stages. The first stage is the dense point cloud reconstruction using Visual SFM. This is the stage where all the information gathered from the photographic data will be processed and used to create a match between every photo and the resulting dense point cloud of our object. The second stage will be to use MeshLab to manipulate the output data from VSFM and clear all spurious information, reconstruct the 3D mesh and transfer color from the vertexes to the mesh. The end result should be a somewhat fully textured 3D mesh of our object (depending on the amount of photographic information collected) which can later be exported out of MeshLab and used for any desired purpose.

ADVANCED STUDIES PROGRAM IN COMPUTATION APPLIED TO ARCHITECTURE, URBAN PLANNING AND DESIGN

3D SCANNING (2C / 3C)

NOV2013

| 2

STAGE 1 DENSE POINT CLOUD RECONSTRUCTION USING VISUAL SFM

1 | PHOTOGRAPHIC SURVEY The first step was to take several photos around the object, trying to capture as many possible angles of the object as possible. While doing this, we had to be careful and make sure that the photos overlapped each other slightly. We can see the process as being similar to creating a panorama photo where each photo taken has to overlap the previous one. This is done so that part of a photo is used as a reference for the proper merging of the next one and so on.

2 | IMPORTING DATA TO VISUAL SFM AND MATCHING The third step was to import all the images into Visual FSM and start the Computing Missing Pairwise Matches process. The result of the matching can be viewed in the View Matching Matrix (fsM > Pairwise Matching > Show Match Matrix). What we see in this matrix are the levels of matched information that Visual SFM was able to establish between our photos. The darker red colors mean a deeper level of matching while the lighter yellow colors mean a lowest level of matching. White areas indicate no matching whatsoever.

FIG.2 | Pairwise matching and respective matching grid 3 | COMPUTING SPARSE 3D RECONSTRUCTION AND DENSE RECONSTRUCTION The next step is to compute the sparse 3D reconstruction from the matching done in the previous step. What we see here happening is Visual SFM taking all the information gathered from our photos and respective matching and calculating the position of each point, relative to a coordinate system, thus creating the point cloud of our object. To conclude this step we run the dense reconstruction (fig. 3), choosing a browser location where we want the date to be stored. In that destination there will be a folder with the name “Models” containing three files, one of which has the extension .ply. This is the file that we’ll use for the next stage in MeshLab.

FIG.3 | The sparse 3D reconstruction and dense reconstruction

ADVANCED STUDIES PROGRAM IN COMPUTATION APPLIED TO ARCHITECTURE, URBAN PLANNING AND DESIGN

3D SCANNING (2C / 3C)

NOV2013

| 3

STAGE 2 3D RECONSTRUCTION AND TEXTURING USING MESHLAB

1 | IMPORTING TO MESHLAB AND CLEANING SPURIOUS DATA After opening MeshLab we need to import the .ply file created in the previous stage (File > Import

Mesh). In the main window we now see the dense point cloud reconstruction of our object including the respective color of each point. What we also see is information that needs to be cleared out first before moving on to the mesh reconstruction. Often when surveying an object, regardless of the methods used, we’ll end up with extra information that needs to be removed first, whether because it’s unwanted or not relevant to the case we’re studying. A point cloud needs, therefore, to be cleared of any such spurious information that might conflict with what we’re trying to achieve. In this case we’re only interested in the stone medallion itself so everything else around it can be erased (fig. 4).

FIG.4 | Imported point cloud and selection of spurious points 2 | MESH RECONSTRUCTION USING POISSON After all spurious information has been deleted we now move on to recreating a 3D mesh out of our points using the Poisson method for reconstruction (Filters > Remeshing, Simplification and

Reconstruction > Surface Reconstruction: Poisson). Because setting the parameters for the mesh reconstruction can be a process of trial and error (sometimes ending with the hardware crashing) we can choose to save the mesh now so that, if we need to return to a previous stage, we don’t have to go through the cleaning process all over again (File > Save Mesh As). Here the only thing we need to be careful with is making sure we check the “Normal” box - mesh reconstruction methods require points to have normals assigned to them. Here we can also choose Binary Encoding which will reduce the file size but at the cost of the information being encoded and difficult to process afterwards if needed. After several tries, I found that the parameters that work best in this case was an Octree Depth of 12, a Solver Divider of 8, 2 samples per Node and a Surface Offsetting of 1. The final result can be seen in fig. 5. To accentuate the geometry slightly an ambient occlusion was calculated at this point (Filters > Color Creation and Processing > Ambient Occlusion – Per Vertex).

ADVANCED STUDIES PROGRAM IN COMPUTATION APPLIED TO ARCHITECTURE, URBAN PLANNING AND DESIGN

3D SCANNING (2C / 3C)

NOV2013

| 4

FIG.5 | Mesh reconstruction using the Poisson method 3 | TRANSFERING COLOR TO THE MESH The reconstructed mesh is currently a textureless mesh so the next step will be to transfer the color information from the point cloud to the mesh (Filters > Sampling > Vertex Attribute Transfer). Here, the point cloud needs to be set as source and the created mesh as the target. The end result (fig. 6) is a partially colored mesh. Parts of the mesh are left without color because there was not enough information in the points cloud to color those areas. It’s now up to us whether to leave the mesh as it is or to go back to the survey and try to photograph the object from the missing angles. However, for several reasons, this may not be always possible to do and so one solution might be to actually erase the parts of the mesh which have no color information. The next step will show a possible method of doing this.

FIG.6 | Mesh with transferred color

ADVANCED STUDIES PROGRAM IN COMPUTATION APPLIED TO ARCHITECTURE, URBAN PLANNING AND DESIGN

3D SCANNING (2C / 3C)

NOV2013

| 5

4 | MESH EDITING Sometimes the object we’re surveying is not always accessible through every angle. In this case, the back of the medallion was obstructed by a wall and so it was not possible to acquire any photos from that side. Because of this, MeshLab is not capable of meshing that side properly and no color information is assigned to it (fig. 7). At this stage we might want to further edit the mesh by deleting any spurious parts of the mesh which have no information. We can choose to do this in another 3D modeling software but if we choose to use MeshLab the process will be similar to that of deleting spurious points only this time we are selecting faces and not points. Since these areas have no color information (are colored white) probably the easiest way to select them is to use a selection filter that selects every face painted with a specific color, white in this case (Filters > Selection > Select Faces by Color).

FIG .7 | Vertex and edge selection filtered by color

FIG .8 | Mesh with erased edges and vertexes

ADVANCED STUDIES PROGRAM IN COMPUTATION APPLIED TO ARCHITECTURE, URBAN PLANNING AND DESIGN

3D SCANNING (2C / 3C)

NOV2013

| 6

_CONCLUSION Using the methods described in this article we now know how to resort to photographical survey methods to reconstruct a textured 3D mesh of an object out of point clouds. The example here show was of a small stone medallion but the methodology of surveying other objects or spaces is similar. It starts with the proper acquisition of all the necessary information first, photos in this case, which will then be used to produce a point cloud of our case study. This point cloud is then worked on further in order to acquire a clean and colored 3D mesh that can be exported and used for in any desired way. In this article we didn’t look at other operations that sometimes need to be conducted to reach a better end result. Some objects may require transformations to be applied, like scaling and rotation, have the point cloud downsampled, and, when we are working with more than one point cloud, these may need to be aligned and merged together. It is therefore important that the reader understands that the process is not always as straightforward as shown here. In the following article we will tackle the use of Terrestrial Laser Scanner for surveying a site, where the process of aligning several point clouds and the manipulation of transformation matrixes will be exemplified in greater detail.

FIG .9 | Depiction of the entire process and comparison between the final result and the original object

Afonso Maria de Castr

This work was conducted

Advanced Studies Program in Computation Applied to Architecture, Urban Planning and DesignFaculdade de Arquitectura

Afonso Maria de Castro Fernandes Abreu Gonçalves, n.º 20130528

http://home.fa.utl.pt/~20130528

This work was conducted for the 3D Scanning course

Advanced Studies Program in Computation Applied to Architecture, Urban Planning and Design

Faculdade de Arquitectura – Universidade de Lisboa 2013 / 2014

20130528

Advanced Studies Program in Computation Applied to Architecture, Urban Planning and Design