Optimizing the use of digital airborne images

for 2.5D visualization

A.S. Sima & S.A.W. KayAgriculture and Fisheries Unit, Institute for Protection and Security of the Citizen,Joint Research Centre of the European Commission, Ispra (Va), Italy

Keywords: monoscopic rendering, visualizations, photogrammetry

ABSTRACT:Monoscopic virtual representations of 3Dgeometries are rapidly becomingimportant products of many databases and software applications. Many GIS tools – evenfreeware, such as Google Earth – permit the visualization of city planningmodels as wellas landscapes derived from 3D geometries (digital surface models draped with imagery,called 2.5D visualization).We propose that off-nadir parts of vertical imagery – typicallyignored after the orthorectification process – provide us systematically with much datathat can be used to optimize the 2.5D rendering process.We attempt to use nominally vertical imagery, collected under typical photogrammetric

conditions, to improve the rendering of vertical surfaces and facades. Results show imagequality at least equivalent to state of the art 2.5D rendering where standard “most nadir”vertical orthoimages are used.

1 INTRODUCTION

Virtual views of rural or urban landscapes presented in 2.5D viewing – that is,monoscopic rendering of imagery draped over a detailed surface model – came of age in2005.On-line services such asGoogle Earth have placed some level of 2.5D capability ondesktop computers, albeit still at a fairly generalized level of texture detail, with freesoftware and standard Earth observation imagery (satellite, airborne).However, all state of the art software and service providers of this nature use Earth

observation data that has been orthorectified, i.e. geometrically corrected to someprojection system and sampled so as to choose the most nadir viewing image available.Nevertheless, more and more imagery is acquired with multiple views – sometimes asmany as 6 or more – that are rarely used in the final orthoimage product or 2.5D renderedviewing. These data offer a potentially rich source for the optimization of textures used inrendering of any image products, either orthoimages or 2.5D rendered views.The products – 2D orthoimages with near-perfect geometry, and 2.5D visualizations

using the highly detailedDSMs – are impressive, and have quickly become economicallycompetitive data sources for large scale urban applications such as land registry updating.Fully orthorectified imagery flight plans, however, produce an enormous amount of

redundant data. Flightswith80%forwardand60%sidelapmayproduce asmanyas15pixelsfor a given point on the ground. Furthermore, the restriction of using only the most nadir

609

GeoInformation in Europe, M.A. Gomarsca (ed.)

�2007 Millpress, Netherlands, ISBN 9789059660618

viewing pixels causes 2.5D rendering to be “smeared” over vertical surfaces (perpendicularto the orthogonal projection plane), such as building facades. We propose instead to exploitthe remaining data, and instead of using only (near) nadir viewing imagery as a data input, tochoose those pixels with an optimized geometry for the rendering process. In our approach,the optimizedpixel is defined as the pixel corresponding to theviewanglemost orthogonal tothe DSM facet that models the true object surface. Figure 1 presents a simplified schema forchoosing the optimal pixel from a set of three images.The studywe present here is more aligned to a standard photogrammetric approach, by

contrast with other work in the computer vision field. We attempt to use nominallyvertical imagery, collected under typical photogrammetric conditions, to improve therendering of vertical surfaces and facades. Image orientation is achieved only through theuse of a precise camera model (including metric calibration information) and camerapositions are provided by the aerotriangulation of a photo block. TheDSM is furthermorealso created using a standard photogrammetric approach and is of a qualitycommensurate with that used for 2D orthoimage production.

2 STUDY DATASET

The images used in this studywere collected inMaussane lesAlpilles, France, on the 24thofMay 2005 byAerodata International Surveys (Deurne, Belgium).AVexcelUltraCamDdigital frame camera was used to collect 80 images in 5 strips, with 80% forward overlapand 60% side overlap.

Figure 1. Simplified schema demonstrating the process of optimizing pixel selection.

610 A.S. Sima & S.A.W. Kay

The flight plan was essentially of classical photogrammetric form (Graham and Koh2002), with nominally vertical images and a flying height of around 5000 m aboveground level. The data used for prototype development were four pansharpenedmultispectral image frames (henceforth termed candidate images), with a groundsampling distance of approximately 0.5m. Exterior orientation ancillary data derivedfrom a standard aerotriangulation process were also made available, comprising thecenters of projection (in UTM projection zone 31N, WGS-84 ellipsoid), camerarotation angles, as well as the calibration report of UltraCamD (Vexcel serial no.18)camera.The terrain model used in this exercise was a digital surface model prepared by

ISTAR S.A. (Sophia Antipolis, France) with a 2m raster cell size. It was extracted(using spatial autocorrelation techniques) from a Leica Geosystems ADS40 airbornesensor image set, acquired 14th of May 2003. The flight altitude was also approximately5000 m with a GSD of 0.5m. The vertical accuracy of this DSM was assessedindependently (Spruyt and Kay 2004) and as RMSEz, on well defined points, wascomputed to be 0.34m.

3 ALGORITHM OPTIMIZATION

The methodology can be divided into four steps:

1. Absolute orientation of candidate image arrays: The main goal of this part is tocalculate coordinates of each pixel of each candidate image frame in the project globalcoordinate system (UTM, zone 31N, ellipsoid WGS-84);

2. DSM processing: The DSM is reprojected back into the candidate image plane; eachtriangular facet of the DSM is represented as an irregular network in the candidateimage space.

3. The information concerning the DSM is used to determine which candidate imageframe is best suited for rendering into the 2.5D scene;

4. Finally, the image rendering is undertaken in a specialized visualization softwarepackage (Reconstructor) using the virtual image files.

3.1 Absolute orientation

Each pixel on the candidate image frame is assigned geodetic coordinates in the worldcoordinate system, using the classical colinearity equation, interior and exteriororientation parameters.In a typical exterior orientation model (built using at least two images), the scale

coefficient l describes the difference between the model and terrain scaling. In our(single) image exterior orientation, l is always equal to 1; there is no change ofscale, because the goal is to know the geodetic coordinates of each image pixel onthe image plane and then be able to rebuild the image geometry at the moment ofacquisition.

611Optimizing the use of digital airborne images for 2.5D visualization 611

3.2 DSM processing

In this step we find a correspondence between the DSM and the candidate images usingexternal orientation parameters. The DSM array (all pixels) is reprojected on the imagesplanes, centre of this projection is identical with the centre of projection of each ofcandidate images.The reprojection of the DSM array to the plane has some disadvantages. The

reprojection is calculated for all the DSM cells from array, even those which are notvisible from the center of projection (for example, on facets facing away from the cameraposition or occluded from view by tall objects). These obstructed pixels need to beidentified and eliminated. We implemented a simple algorithm that evaluates thereprojected position of each DSM cell on the image plane, and determines if adjacentpixels are obstructed.Finally, the now irregular grid of reprojected DSM data must be networked as

triangular facets. Only those pixels visible from the center of projection of each exposureare selected for this process.When for all vertices of triangles on the ground corresponding coordinates on the

image plane are found, texture extraction becomes possible.

3.3 Integrating DSM and candidate images

To extract the texture from candidate imagewe have to find position of all vertices in eachtriangle on the ground in the pixel coordinate system [column, row] of the image. This isdone by searching the position of all reprojected vertexes in the regular array of pixelscoordinates created in the Step 1.To determine which candidate image is the best choice for texturing any individual

DSM triangular facet, it is necessary to compare geometry of acquisition (view anglefrom the center of projection to the center of the triangular facet) of each candidate imagewith the normal to this triangle.The best texture is the candidate image pixel with a view angle most orthogonal to the

DSM facet, so the angle between the normal vector and the ‘view on triangle’ vectorshould be as close to 180� as possible (see Figure 1).

3.4 Image rendering

The algorithm was implemented in Interactive Data Language (RSI). Use was made of aLinuxoperating systemon aPCplatformdue to better possibilities ofmemorymanagement,a critical factor with the processing of the large datasets (thewhole imagematrix consists of11500� 7500 pixels, double-precision data). Processing times on a 2.4GHz machine werequite demanding, in the order of 6 h for image sets of only 800� 800 pixels.The resulting output files from our prototype algorithm are then processed using a

script written in MatLab programming language (G. Bostrom, pers. comm.), whichprepares files readable for the visualization software Reconstructor (Topotek 2005). Theexamples presented in the paper all use this software for rendering the 2.5D visualization.

612 A.S. Sima & S.A.W. Kay

4 RESULTS

The prototype algorithm was able to correctly select candidate images from the testdataset, and make virtual images available for rendering in the visualization softwareenvironment. Examples of the test results can be viewed in Figures 3 and 4 below.Figure 3 showsa typical rural landscape from theMausannedataset.The raisedobjects in

theDSMhave been rendered very accurately with the appropriate image data. The texturesappear to be very compatiblewith adjacent pixels derived from different candidate images,and no special artifacts are immediately visible in the 2.5D virtual image.Figure 4 shows a pseudo-colored image, with each of four colors representing one of

the input candidate image frames. It can be easily seen that raised objects (hedges, copses)have acquired texture from up to four images, which is rendered in accordance with theaspect and slope of the DSM fact being processed.

DSM Build an imagearray

Interiororientation

Externalorientation

Step 2

Step 1

Step 4

Image

Input data

Output data

LEGEND:

Image size: 12500 x 7500 pixPixel size: 9 µ

Focal length: 101,4 mmDistortion values

(X0, Y0, Z0, ω, ϕ, χ)External orientation parametersSelect the part

of DSM commonfor all candidate

images

Reproject the DSMonto image plane

(pinholeprojection)

Eliminate

Triangulate the data

search

column, row

Rendering,Reconstructor

Texture ID

Determinate theOptimum candidateimage to use as a

texture for the triangle

obstructed pixels

X, Y, Z X, Y, Z

Step 3

Figure 2. Simplified schema demonstrating the process flow of optimizing pixel selection.

Optimizing the use of digital airborne images for 2.5D visualization 613

Although these images show the successful execution of the optimization algorithm, thegeneralization due to the DSM cell size (2m) still causes a large degree of smoothing in therendered image. Furthermore, candidate images were not chosen optimally with respect tooverlap, resulting in a set of candidate images that in fact were near-vertical viewing.

Figure 3. The optimized 2.5D view sample: texturing with real images. The image and view point

is identical to Figure 5.

Figure 4. In this graphic, each color is a substitute of the texture from a one of the four candidate

images.

614 A.S. Sima & S.A.W. Kay

In order to try and overcome these circumstances – aswell as to test the implementationon an independent set of candidate images not used for the development – a second dataset(Brasschaat, Belgium), acquired by Aerodata International, was processed using theprototype algorithm implementation. The input data were similar to the original test site,acquired with the same digital camera; however, image scale was larger (20 cm groundsampling distance) and with a 1m DSM (autocorrelation on the original imagery).Moreover, the scene is composed of an urban environment, with numerous buildings aswell as landscape objects such as trees and bushes. The result of the 2.5D rendered imagecan be seen in Figure 5.Again, the algorithm performs satisfactorily with respect to the general selection and

positioning of image texture.However, theDSMis again rather smooth, causing the detailof buildings to be disrupted. Although some facades appear to be rendered with texturecolors differing from those used for building roofs, in fact there is still very little data onvertical surfaces available for the algorithm in such a flight configuration.

5 CONCLUSIONS

We have demonstrated here a proof of concept algorithm for the optimization of imageryfor 2.5D visualization.The algorithm has been implemented in a standard image processing tool and is able to

operate on frame camera imagery, such as the Vexcel UltraCamD. Results show imagequality at least equivalent to state of the art 2.5D rendering where standard “most nadir”vertical orthoimages are used.Limitations of the results so far are due to the rather vertical nature of the datasets

tested, rather than any specific issues with the algorithm itself. The visualization resultsare still inferior to datasets where either terrestrial imagery has been used for facades (forexample, Bostrom et al. 2004), or where specially acquired oblique airborne imagery isintegrated in the rendering process (Fruh et al. 2004).We believe that our results would be

Figure 5. Result of the supplementary trial with higher resolution dataset (Brasschaat, Belgium).

Optimizing the use of digital airborne images for 2.5D visualization 615

improved if imagery were acquired with a wider field of view and from a lower flyingheight; we conclude that the algorithm would nonetheless perform effectively with suchdata since there is no fundamental change in the geometry of image acquisition.Additionally, a more refined DSM would significantly improve visualization results.Many areas for optimization of the algorithm implementation remain. Data processing isslow due to lack of triangulation optimization, requiring around 6 h to prepare theoptimized 2.5D visualization of an area of 800 · 800 pixel image sample. We considerthat after further program optimization, processing time would be cut by 2/3rds.

ACKNOWLEDGEMENTS

We would like to thank: Fred Hagman and Willem Philipse (Aerodata InternationalSurveys, Deurne, Belgium), Gunnar Bostrom and Vitor Sequeira (Nuclear SafeguardsUnit, JRC Ispra, Italy), Giovanni Banchini (CGR S.p.A., Parma, Italy), Arthur Rohrbachand Udo Tempelmann (Leica Geosystems AG, St. Gallen, Switzerland).

REFERENCES

Bostrom, G., Fiocco, M., Puig, D., Rossini, A., Goncalves, J.G.M. & Sequeira, V. 2004. Acquisition,Modelling and Rendering of Very Large Urban Environments. Proc. of the Second InternationalSymposium on 3DData Processing, Visualization and Transmission, Thessaloniki, Greece, Sept6–9, 2004: 191–198.

Devos, W., Kay, S. 2005. Assessing Geometric Accuracy of Ortho-imagery: Towards ProductSpecification ofFullyOrthorectified Imagery forUrbanAreas,GIMInternational 19 (6): 14–17.

Fruh, C., Sammon, R. & Zakhor, A. 2004. Automated Texture Mapping of 3D City Models WithOblique Aerial Imagery. Proceedings of the Second International Symposium on 3D DataProcessing, Visualization and Transmission, Thessaloniki, Greece, Sept 6–9, 2004: 191–198.

Graham, R., Koh, A. 2002. Digital Aerial Survey: Theory and Practice, CRC Press, Boca Raton,Florida.

Spruyt, P., Kay, S. 2004. Quality assessment of digital airborne orthoimagery: A test study usingthe Leica Geosystems ADS40, GIM International 18 (6) 35–37.

Topotek, 2005. RECONSTRUCTOR: The laser data processing package, URL: http://www.topotek.it/modules.php?name=Content&pa=showpage&pid=14 Topotek, Consortium forTechnology Innovation, c/o Engineering–University of Brescia, 38, Branze, I–25123 Brescia,Italy (last date accessed: 22 Dec 2005).

616 A.S. Sima & S.A.W. Kay