D8.8.1REPORTONTHEIQMULUSPROCESSINGCONTEST–YEAR1

DeliverableD8.8.1

Circulation: RE‐ Restricted

Leadpartner: CNR‐IMATI‐GE

Contributingpartners: CNR‐IMATI‐GE,TUDelft,UCL,IGN,UBO,FOMI,

Authors: SilviaBiasotti,SimonePittaluga (CNR‐IMATI‐GE);RoderikLindenbergh,BerilSirmacek(TUDelft),JanBoehm(UCL);ClémentMallet,NicolasPaparoditis(IGN),RomainCancouet(UBO),DanielKristof[FOMI]

QualityController: EwaldQuak(SINTEF)

Version: 1.0

Date: 31.10.2013

D8.8.1 IQmulus(FP7‐ICT‐2011‐318787)

1

©Copyright2013:TheIQmulusConsortiumConsistingof

SINTEF STIFTELSENSINTEF,DepartmentofAppliedMathematics,Oslo,Norway

Fraunhofer FraunhoferInstituteforComputerGraphicsResearch,Darmstadt,Germany

CNR‐IMATI‐GE Institute for Applied Mathematics and Information Technologies of theNationalResearchCouncil(CNR‐IMATI),Genova,Italy

MOSS M.O.S.S.ComputerGrafikSystemeGmbH(MOSS),Munich,Germany

HRW HRWallingfordLtd(HRW),Wallingford,UK

FOMI Hungarian National Mapping and Cadastral Agency (FÖMI), Institute ofGeodesy,CartographyandRemoteSensing,Budapest,Hungary

UCL University College London (UCL), Research centre for Photogrammetry, 3DImagingandMetrology,London,UK

TUDelft Delft University of Technology (TUDelft), Department of Earth and ClimateSciences&Man‐MachineInteractionGroup,Delft,TheNetherlands

IGN Institut National de l’Information Géographique et Forestière (IGN), Paris,France

UBO Université de Bretagne Occidentale (UBO), European Institute for MarineStudies,Brest,France

Ifremer L’Institut Francais de Recherche pour l´Exploitation de la Mer (Ifremer),Brest,France

Liguria RegioneLiguria,Genova,Italy

Thisdocumentmaynotbecopied,reproduced,ormodifiedinwholeorinpartforanypurposewithout written permission from the IQmulus Consortium. In addition to such writtenpermissiontocopy,reproduce,ormodifythisdocumentinwholeorpart,anacknowledgementoftheauthorsofthedocumentandallapplicableportionsofthecopyrightnoticemustbeclearlyreferenced.

Allrightsreserved.

Thisdocumentmaychangewithoutnotice.

IQmulus(FP7‐ICT‐2011‐318787) D8.8.1

2

DOCUMENT HISTORY 

Version1  Issue Date  Stage  Content and Changes 

0.1  Oct72013 Draft  Initialdraft0.2  Oct122013 Draft   

0.7  Oct.222013  Draft   

1.0  Oct.312013    VersiontobesubmittedtotheProjectOfficer

EXECUTIVE SUMMARY

ThedeliverableD8.8.1isrelatedtotheIQmulusTask8.5,describedasfollowsintheDescriptionofWork(DoW):

Focusof thetaskwillbethedisseminationof IQmulusresultsondataprocessingbymeansofsettingupandrunningyearlyconests,open to thewholescientificcommunityaswellasusergroups,onresearchissueswhicharerelevanttotheWP4areaofwork.Suchbenchmarkingandcontests have been widely used in the multimedia domain (e.g., TRECVID, IMAGECLEF), andrecentlyinthe3Dretrievalarea(SHREC)asakeymeanstohighlightandrecognizesignificantadvancesinresearchandensurereproducibilityofresults.

Thisreportdescribesthefirstyear’sactivityoftheIQmulusprocessingcontest,theselectionofthemostinterestingtasks/servicesproposedascontesttracks,andthelistofparticipantsintheproposedchallenges.Inaddition,welistthepanelofexperts,selectedamongthepartnersandinvolvingscientistsoutsidetheconsortiumthathavebeenchosentosupporttheplanningofthecontest,establish theprocessingtasks tobebenchmarkedandselect theappropriatedataset,performancemeasuresandprocedurestorealizethevalidation.

1 Integers correspond to submitted versions 

D8.8.1 IQmulus(FP7‐ICT‐2011‐318787)

3

Tableofcontents

Executivesummary..........................................................................................................................2 

1  Introduction...............................................................................................................................5 1.1  Related benchmarks and events ................................................................................................. 5 

1.2  Advisory board ............................................................................................................................ 6 

2  Timelineofthecontest...........................................................................................................6 

3  Thecontestinfrastructure....................................................................................................7 3.1  The Collage Authoring Workbench ............................................................................................. 7 

4  Thefirstyear’stracks.............................................................................................................8 4.1  Registration of Point Clouds ........................................................................................................ 9 

Datasets provided by IGN (four): ......................................................................................................... 9 

Datasets originating from airborne Lidar scans of Regione Liguria (two): ....................................... 11 

4.2  Feature extraction (drainage basins from a DEM) .................................................................... 11 

5  Participantsofthecontest.................................................................................................12 5.1  Registration of Point Clouds ...................................................................................................... 12 

5.2  Feature extraction (drainage basins from a DEM) .................................................................... 12 

6  Evaluationmethodology.....................................................................................................13 6.1  Registration of Point Clouds ...................................................................................................... 13 

6.2  Feature extraction (drainage basins from a DEM) .................................................................... 13 

7  Futureplans.............................................................................................................................14 

8  References................................................................................................................................15 

IQmulus(FP7‐ICT‐2011‐318787) D8.8.1

4

ListofFigures

Figure1AscreenshotofthemainpageoftheCollageAuthoringWorkbench........................................8 

Figure2TwodistinctairbornedlidarscannersofurbanareaofAmiens,France.Left:mobilemodel,Right:referencemodel.Coloursrepresentthez‐values...........................................................9 

Figure3OneofthetwopointcloudsoftheParis1_MMSdataset.ThecolourscorrespondtotheZ‐values.....................................................................................................................................................................10 

Figure4picturewiththetwopointcloudsoftheParis2_MMsdataset.onecolourforeachdataset........................................................................................................................................................................10 

Figure5twoviewsoftheparis3_MMSdataset.left:topview,Z‐colored;Right:obliqueview,GPS‐timecolored.............................................................................................................................................................11 

Figure6Left:ThereferencedatasetofthecityofGenova,Right:oneofthepointscloudsoftheVernazzadataset....................................................................................................................................................11 

Figure7Thethreedatasetsselectedfortheextractionofthedrainagebasins...................................12 

D8.8.1 IQmulus(FP7‐ICT‐2011‐318787)

5

1 INTRODUCTION

Enabling reproduction and evaluation of an algorithmoutcome is an acute issue in computerscience that rapidly grows with the increase of the demand for efficient and computablesolutions inmany fields. In thiscontext, thecreationofstandarddatasetswithagroundtruthand a “quality label” paves the road to a fair evaluation of the existing technologies and theidentificationofnewdirectionsofresearch.

TheIQmulusprocessingcontestwillcomplementandcooperatewiththeexistingbenchmarkingactivities within its sphere of operation so that opportunities and technical gaps can beidentified. It is a specific goal of the IQmulus contest to become a reference for the 3D pointcloud processing, proposing examples from the Maritime Spatial Planning and the LandApplications for Rapid Response and TerritorialManagement scenarios. Indeed, IQmuluswillcombine in thesamecontest3Dpoint clouds,generatedbyLidarsensorsand/orstereovisionandwillanswertotheincreasingdemandofstandardbenchmarksingeospatialdataprocessingandfosterthecombinationofheterogeneousdata.

Theresultsof thecontestswillbemadepubliclyavailableand inparticularareplanned tobeequippedwithexecutablecodesoas tomaketheparticipation in thecontestsmoreattractiveforscientists.ThepartnersinvolvedinWP4onProcessingServicesareexpectedtoparticipatein thecontestwith thesubmissionof thealgorithmsandmethodsdeveloped in theproject, inrespectoftheIPRrulesandrestrictionsasidentifiedintheConsortiumAgreement.

Thisreportdescribesthefirstyear’sactivityoftheIQmulusprocessingcontest,theselectionofthemostinterestingtasks/servicesproposedascontesttracksandthelistofparticipantsintheproposedchallenges.Inaddition,welistthepanelofexperts,selectedamongthepartnersandinvolvingscientistsoutsidetheconsortiumthathavebeenchosentosupporttheplanningofthecontest, establish theprocessing tasks tobe benchmarked and select the appropriatedataset,performancemeasuresandprocedurestorealizethevalidation.

1.1 RELATED BENCHMARKS AND EVENTS

Atremendousoutbreakofsimilarbenchmarkingactionsandcontestshasbeennoticedforthelast five years in many areas of computer science: they are a key means to highlight andrecognize significant advances in research and ensure reproducibility of results. For instance,benchmarkshavebeenwidelyused in themultimediadomain (e.g., TRECVID (Smeaton et al.,2006,Smeatonetal.,2009,Overetal.,2012), IMAGECLEF(Muelleretal.,2010), inmulti‐viewstereo(Strechaetal.,2008)orinimage‐basedobjectdetection(Everinghametal.,2010).Morerecently,theyhavebeenadoptedintheareaof3Dretrieval(SHREC(Veltkampetal.,2006))andkeypointdetection(Tombarietal.,2013),invisionforimagesequenceanalysisandautonomousnavigation (Geiger et al., 2012, Goyette et al., 2012, Geiger et al., 2013), and in the computergraphicscommunityforsurfacereconstructionfrompointclouds(Bergeretal.,2013).

AninitiativesimilartotheIQmulusprocessingcontesthasstartedin2011undertheaegisoftheISPRS (Rottensteiner et al., 2012). The need of new standard test sites that exploit theavailability of modern airborne sensors led to the creation of a benchmark on urban objectextraction. In particular, both airborne topographic Lidar data andmulti‐view stereo imageryareprovidedtocompareandevaluateresearchalgorithmsforbuildingdetection,treedetectionand building reconstruction in urban areas. In addition, a benchmark dataset on shaperegistrationhasbeenrecentlyproposedinthefieldofroboticresearch(Pomerleauetal.,2013).

IQmulus(FP7‐ICT‐2011‐318787) D8.8.1

6

Theresultsofthecontesthighlightedthatthereisstillplaceforfurtherdevelopmentsinallthescenariosconsidered;inparticularthedetectionofsmallbuildingstructuresandtreesandtheproduction of high‐quality level‐of‐detail buildingmodels. As a further outcome, themethodscurrently at the state of the art donot seem to be able to fully exploit the accuracypotentialinherentinthesensordata.

A recentdevelopment is also the increaseof popularity ofOpenSource libraries.Well‐knownexamplesarePointCloudLibrary(PCL),GDAL,CGALandOpenCV,butthefulllistismuchlongerand ranges from professional well‐maintained to short‐lived, poorly documented code. Thisposesanewchallengeforresearchersandsoftwaredevelopers:towhatextentisitpossible,intermsofperformance, toembedcode fromOpenSource libraries inanewproject.Toat leastpartlyanswerthisquestion,somefunctionalitiesoftheseOpenSourcelibrariesaregoingtobetestedwithintheframeworkoftheIQmulusprocessingcontestaswell.

1.2 ADVISORY BOARD

Toallowalargescientificconsensus,theIQmuluscontestisimplementedincollaborationwiththeresearchcommunityandthesupportofaninternationaladvisoryboard,composedofhighlevelscientificandtechnicalexpertsinthefield.

Forthefirstyear,themembersoftheIQmulusconsortiuminvolvedintheAdvisoryBoardare:

SilviaBiasotti,CNR‐IMATI‐GE JanBoehm,UCL ClémentMallet,IGN RoderikLindenbergh,TUDelft

ThemembersoftheAdvisoryBoardthatareexternaltotheIQmulusprojectare:

HamishCarr,VisualizationandVirtualRealityGroup,UniversityofLeeds,UK; DebraLaefer,SchoolofCivil,Structural&EnvironmentalEngineering,UniversityCollege

Dublin, Ireland. Debra Laefer is the recipient of the European ERC Starting Grant“RETURN:RethinkingTunnellinginUrbanNeighbourhoods”,2013‐2017.

2 TIMELINE OF THE CONTEST

In the following,we briefly summarize the different steps of the preparation of the first yearIQmuluscontest.

Jan2013:Creationofthemailinglistofcontactsforthecontest

May 2013: Start of the activity: discussions of the actions to be carried out the first year,challengestobemet,analysisoftheexistingdatasetsandcomparisonwithotherbenchmarks,analysis of the possible ground‐truth, identification of themost promising tracks (in termsofparticipationandpossibleground‐truth)

30June2013:JointsubmissionoftheIQmuluscontesttotheISPRSworkshopLaserScanning2013

July 2013: Identification of the tracks to be implemented in the first year’s contest, namelyregistrationandfeatureextractionofdrainagebasins

D8.8.1 IQmulus(FP7‐ICT‐2011‐318787)

7

7Aug2013:Circulationofthefirstcallforparticipation

20Sep2013:Circulationofthesecondcallforparticipation

Oct2013:Thecontestisrunningandtheresultsareanalysedandevaluated

Nov2013: The resultswill be presented at the ISPRSworkshop Laser Scanning 2013, 11‐13November2013,Antalya,Turkey

3 THE CONTEST INFRASTRUCTURE

ThecontestisorganizedonanexistingcomputationalplatformhostedbyCNR‐IMATI‐GE.SuchaplatformisdistinctfromtheoverallIQmulusinfrastructureandisaCPU‐clusterunitrelatedtothe Visualisation Virtual Services (VVS) offered by the FP7‐INFRASTRUCTURES projectVISIONAIR(2011‐2015),forthepreparationofvirtualscenesandexperimentsforanygraphicaland Virtual Reality settings. The VVS is conceived as an operational andweb‐based resourcerepository, which involves the integration of a Shape Repository, a Tool Repository and anOntologyandMetadataRepository,togetherwithanadvancedSearchFramework.Theoutcomeofthecontest,beingevaluatedonaninfrastructureindependentoftheIQmulusone,willbeusedtoestimatethegainthatisobtainedbyrunningtheservicesontheIQmulusinfrastructureitself.

Tohost the IQmulusProcessingcontest, theVVS isupgradedaugmenting itsdatastorageandprocessingcapabilities.Theservices forrunning thebenchmarkingandevaluationproceduresare implemented using the Collage Authoring Workbench, http://collage.ge.imati.cnr.it,developedbyAcademicComputerCenterCyfronetAGHinCracowinPoland,whichallowsthedocumentationandthelong‐termpreservationoftheoutcomeofthecontests.

3.1 THE COLLAGE AUTHORING WORKBENCH

The Collage Authoring Environment is a framework for collaborative preparation andpublication of so‐called computational experiments that can back executable papers. TheenvironmentenablesresearcherstoseamlesslyembedchunksofexecutablecodeanddataintoscientificpublicationsintheformofCollageitems,andtofacilitaterepeatedexecutionofsuchcodes on underlying computing and data storage resources. Using the Collage AuthoringWorkbenchscientistscandevelopexecutablepapersandgenerateitemsthatcanbeembeddedinto a web site. The Collage team piloted the Collage Environment for a Special Issue ofComputers & Graphics, published in May 2013. Elsevier published this Special Issue ofComputers & Graphics on ScienceDirect and integrated computational elements created byauthors using Collage into the on‐line article. For more information and links to the articlespleaseseewww.elsevier.com/executablepaper.

The Collage Authoring Environment was originally developed in support of research teamslinked to the PL‐Grid project. It is based on the GridSpace2 platform thatwas targeted at anunrestricted range of communities. GridSpace2 was designed and is still developed by theDistributedComputingEnvironmentsTeam,apartofAcademicComputerCenterCyfronetAGHin Cracow, Poland. The Collage Authoring Environment, along with a publishing serverprototype, won first prize in the Executable Paper Grand Challenge organized as part of theInternational Conference on Computational Science 2011 in Singapore (Ciepiela et al., 2013,Ciepielaetal.,2012,Nowakowskietal.,2011).

IQmulus(FP7‐ICT‐2011‐318787) D8.8.1

8

Figure 1 shows a screenshot of the initial page the participants findwhen connecting to thecontest infrastructure. The infrastructure is accessible via https protocol at the web addresshttp://collage.ge.imati.cnr.it/workbenchandatutorialwithexamplesonitsusageareavailableathttp://collage.ge.imati.cnr.it/tutorials.

Currently, the operating system used in the Collage Authoring Workbench adopted in theIQmulusProcessingContestisLinuxUbuntu12.04.There, the participantsmay run their code and executables using the classic Unix bash shell(http://www.hypexr.org/bash_tutorial.php#emacs),theWindowsfiles.execanberunusingtheenvironment “wine” (http://www.winehq.org/) and the interpreted language Octave 3.2.1(http://www.gnu.org/software/octave/) is available to runMATLAB‐like code. Also plots arepossible using the gnuplot package (http://www.gnuplot.info/). To support geospatial dataprocessing,additional libraries,suchasPCL,GDAL,MPI,VDK,FLANN,Boost,etc.,areavailableandalreadyinstalledintheVVSinfrastructure.

FIGURE1ASCREENSHOTOFTHEMAINPAGEOFTHECOLLAGEAUTHORINGWORKBENCH

4 THE FIRST YEAR’S TRACKS

Thecontestenablesreproducibleevaluationthroughthecreationofstandardizedbenchmarksand evaluationmethodologies. In the first year, two challenges have been identified as acuteissuesingeospatialdata:i)theregistrationofsetsofpointcloudsandii)featureextraction(inparticulartheextractionofdrainagebasinsfromaDEM).

Inthefollowing,wedescribethetwosetsoftestsproposedinthecontest,distinguishingthemaccordingtothechallengeproposed.

D8.8.1 IQmulus(FP7‐ICT‐2011‐318787)

9

4.1 REGISTRATION OF POINT CLOUDS

Sixdatasetshavebeenselectedtoevaluatethepropertiesof themethodsandareprovidedtotheparticipants.Thedatasetsaremadeavailableinthefolder“Registration”inthehomepageofeach participant on the Collage AuthoringWorkbench. The datasets answer to problems thatariseindataregistration,forinstancetheregistrationofpointcloudsacquiredintwodifferentepochs (for instance the Amiens_AirborneLidar dataset); the registration of crowded urbanstreetsacquiredwithamobilevehicle(forinstancethedatasetsParis1_MMSandParis3_MMS);theregistrationofdifferentairborneLidar(for instancethedatasetsoriginating fromRegioneLiguriadata).

DatasetsprovidedbyIGN(four):

Amiens_AirborneLidar: The dataset has been acquired over a dense urban area (city centerwithlow‐risebuildings,Amiens,France)attwodifferentdates,andwithdistinctairborneLidarscanners,resulting intwodifferentgroundpatterns,Figure1.Thefirstacquisitionwas completed in2003 (pointdensityof7.5pts/m²,400,000pts),withaToposys fiberscanner.Spatialsamplingisthereforeveryinhomogeneous(1.5mbetweentwofibers,and0.15mbetweentwomeasurementsofthesamefiber).Thesecondacquisitionoccurredin2008(pointdensityof2pts/m²,90,000pts,oscillatingmirror)withanOptech3100EAdevice(Gressinetal.,2013).

FIGURE2TWODISTINCTAIRBORNELIDARSCANSOFTHEURBANAREAOFAMIENS,FRANCE.LEFT:MOBILEMODEL,RIGHT:REFERENCEMODEL.COLOURSREPRESENTTHEZ‐VALUES.

Paris1_MMS This dataset is acquired with the Stereopolis vehicle, (Monnier et al., 2013,Paparoditisetal.,2012).ThepositionalsystemiscouplingtwoGPSandoneApplanixIMUPOSVL200.TheGPSacquirepositionat1Hz,whiletheIMUhasafrequencyrateof100Hz.ThisIMUhasadriftof3deg/hr,resultinginanaccuracyofabout20cmforaGPSsignallossof1minute(2mfor3minutes).StreetinParis,samechallengesasforParis3_MMS,seeFigures3and5.

IQmulus(FP7‐ICT‐2011‐318787) D8.8.1

10

FIGURE 3 ONE OF THE TWO POINT CLOUDS OF THE PARIS1_MMS DATASET. THE COLOURSCORRESPONDTOTHEZ‐VALUES

Paris2_MMSThesepointcloudsareacquiredwiththeStereopolisvehicle,(Gressinetal.,2013,Paparoditisetal.,2012).ThisdatasetcoversonebuildingfacadeinanurbanareainParis,France. Themobilemapping systemacquired the same area twice on the samedaybutwithatimeshiftofonehour.ARIEGLLMS‐Q120ilidarhasbeenusedforthatpurpose,andorientedtowardstherooftop(pavementandroadsurfacesomitted).Thechallengeshereare:(1)notexactlythesamepartsofthebuildingsaresampledand(2)a3Dshiftbetweenboth point clouds naturally exists, due to the georeferencing process. The point clouddensity is very variable, even within the same point cloud, depending on the angle ofincidence of the variable distance between objects and the MMS. However, the typicaldensityonthefacadesofthebuildingsisnear100pts/m²(around200,000ptsperpointcloud),seeFigure4.

FIGURE 4 PICTURE WITH THE TWO POINT CLOUDS OF THE PARIS2_MMS DATASET. ONECOLOURFOREACHDATASET

Paris3_MMS: This dataset was acquired with the Stereopolis vehicle, (Gressin et al., 2012,Paparoditis et al., 2012). This dataset covers a surface of 0.5 km², with a trajectory of1.5km along several streets. The whole point cloud includes 5 million points. Theacquisitionprocedureofthisdataset isbasedonaRIEGLLMS‐Q120i lidarsensor,whichprovides vertical profiles perpendicular to the platform trajectory. Positioned vertically,the lidar sensor allows acquiring the roads, the sidewalks andmedium‐sized buildings.However,thetopsofthehighestbuildingsmaynotbevisible,dependingonthedistancefrom the sensor to the building. This configuration allows to cover areas with several

D8.8.1 IQmulus(FP7‐ICT‐2011‐318787)

11

interesting objects (street furniture), but also included many moving objects such aspedestrianorcars(Figure5).

FIGURE 5 TWOVIEWSOF THE PARIS3_MMSDATASET. LEFT: TOP VIEW, Z‐COLORED; RIGHT:OBLIQUEVIEW,GPS‐TIMECOLORED.

DatasetsoriginatingfromairborneLidarscansofRegioneLiguria(two):

ThesedatasetsarecreatedfromAirborneLidarscansprovidedbyRegioneLiguriaandownedby the ItalianMinisterodell’Ambiente. Tomake these datasets suitable for a data registrationchallenge, the data have been randomly perturbed with Gaussian noise and rotated withdifferentrotations(varying from0.25rad to0.9rad). Inaddition, theoriginal regulargrid (1‐meterscan)hasbeennon‐uniformlyfilteredusingarandomfilter.

Vernazza:thisdatasetismadeofthreepointclouds, in .plyASCIIformat,originatingfromthesameacquisition(sample23inthesampledatasetsoftheIQmulusproject).Thisdatasetpresents both urban characteristics (buildings and streets) and countryside ones (hillswithterracing‐likeshape),thusshowingahighshapevariability(Figure6,right).

Genova: this dataset ismade of four point clouds, in .ply ASCII format, originating from twodifferentacquisitionsoftheurbancenterofthecityofGenova(Figure6,left).

FIGURE 6 LEFT: THE REFERENCE DATASET OF THE CITY OF GENOVA, RIGHT: ONE OF THEPOINTCLOUDSOFTHEVERNAZZADATASET

4.2 FEATURE EXTRACTION (DRAINAGE BASINS FROM A DEM)

Threedatasetsareproposedforthetrackontheextractionofdrainagebasins.ThethreemodelscomefromtheDEMoftheLiguriaarea,asprovidedbyRegioneLiguria.TheDEMmodelsareingeotiffformatandcorrespondtoasamplingoftheareawithapassof5metres.Thealgorithmsintheliteraturefortheextractionofmedium‐largedrainagebasins(morethan10mq)performverywellandreliably.Thechallengeposedbythesedatasets istheselectionofmedium/smalldrainagebasins.

IQmulus(FP7‐ICT‐2011‐318787) D8.8.1

12

Thechoiceofthethreeareas(Savona‐Albissola,Portofinoanditssurroundings,CinqueterreandLaSpeziagulf)correspondstotwospecificchallenges:

1. Theselectionofsmalldrainagebasins(lessthan10mq,approximativelyaround4‐5mq)isstillanopenproblemandneedsthevalidationonrealcases;

2. Theidentificationofsmallzoneshydrogeologicallyunstablebutwithalargeimpactonthecontext:thelargevariabilityoftheshapeoftheseareasmakesthemveryriskywhenalargeamountofrainfallsinashorttime.Forinstance,theCinqueterreareawassubjecttoadramaticflashfloodthatdamagedalargepartofthisarea.Inaddition,twooftheseareas have a large environmental importance: Portofino and Cinqueterra are bothnaturalparksofnationalrelevance.

ThepicturesofthethreesitesareshowninFigure7.Theintensityofthegreylevelrepresentsthez‐value.

FIGURE7THETHREEDATASETSSELECTEDFORTHEEXTRACTIONOFTHEDRAINAGEBASINS

5 PARTICIPANTS OF THE CONTEST

In this section we list the participants of the contest tracks, with a brief description of thetechniquesusedtoparticipateinthetrack.

5.1 REGISTRATION OF POINT CLOUDS

Sixparticipantshavebeenregisteredforthistrack,threeofthemaremembersoftheconsortiumandthreeareexternalparticipants.Namely:

1. IGN,ICPvariant2. TUDelft,feature‐basedregistration3. UCL,feature‐basedregistration4. FrancoisPomerleau,AutonomousSystemsLab,ETHZürich,usinganICPvariant5. MartinIsenburg,RapidLasso,usingtheLAStoolslibrary6. ZhizhongKang,ChinaUniversityofGeosciences,BeijingCity,China.

5.2 FEATURE EXTRACTION (DRAINAGE BASINS FROM A DEM)

Threeparticipantshavebeenregisteredforthistrack;allofthemaremembersoftheIQmulusconsortium,namely:

1. B.Sirmacek,Dept.ofGeoscience&RemoteSensing,TUDelft.

D8.8.1 IQmulus(FP7‐ICT‐2011‐318787)

13

2. S. Pittaluga, CNR‐IMATI‐GE (with two different algoritms: the TauDEM modulehttp://hydrology.usu.edu/taudem/taudem5/index.html and the GRASS onehttp://grasswiki.osgeo.org/wiki/Hydrological_Sciences)

3. V.PhanHien,Dept.ofGeoscience&RemoteSensing,TUDelftwiththeArcHydromodulehttp://blogs.esri.com/esri/arcgis/2011/10/12/arc‐hydro‐tools‐version‐2‐0‐are‐now‐available/

6 EVALUATION METHODOLOGY

6.1 REGISTRATION OF POINT CLOUDS

Theground‐truthforallthepointcloudschosenforthistrackiswellknown.

Forthedatasetswithrealmisalignments,theground‐truthconsistsofthetransformationmatrixobtainedviathemanualalignmentofthetwopointcloudsfromthebestsolutionretrievedfortheautomaticalgorithmdevelopedforthatpurpose(Gressinetal.,2013).

ForthetwodatasetsderivedfromAirborneLidarscansoftheareasofGenovaandVernazza,theexact transformationmatrix from theoriginaldataset and theperturbedone is knownby theconstructionofthedataset.Theoriginaldatasetwiththerecordofthecorrespondencesintheperturbed one is also known; therefore,we knowwhich point of one dataset corresponds towhichpointintheotheroneandtherotation/translation/perturbationapplied.

Evaluation criteria. The evaluation of the performance of the methods proposed is bothqualitative and quantitave. As quantitative criteria we consider the time expired and thememory occupied by each run. The qualitativemeasures aim at evaluating the quality of theregistrationintermsofthemeanandstandarddeviationofthedistancesoftheregisteredpointsto theircorrespondences in theground‐truthpointcloud;and,whenavailable, in termsof thedifferencebetweenthetransformationmodeldetectedbythemethodandtheground‐truthone.

6.2 FEATURE EXTRACTION (DRAINAGE BASINS FROM A DEM)

The ground‐truth is theofficialmapof thedrainagebasinsownedbyRegioneLiguria. Such amapisvalidatedbyexpertsandcurrentlytakenasthereliablemapofthedrainagebasinsoftheRegione Liguria. To evaluate the methods proposed, a subset of significant features (e.g.,drainagebasinsandbreaklinesthataresignificantfromthegeospatialpointofview)havebeenidentifiedandthefinalscorewilldependonthecorrectidentificationofthesecharacteristics.

To validate the methods participating in the track we consider qualitative and quantitativeparameters like in the registration track. Similarly to the registration track,we compute timeandmemorystorageastwoquantitativeindices,whilethequalityoftheresultsisobtainedbysuperposing the official drainage map with the outputs of the methods and counting theelementsthatcorrespondinthetwomaps,thedeviationofthetwomapsintermsofdifferencesoftheareasofthebasinsanddeviationoftheflowlines.

IQmulus(FP7‐ICT‐2011‐318787) D8.8.1

14

7 FUTURE PLANS

ThesetupandtheyearlyrunofopencontestsarecrucialforthedisseminationoftheIQmulusresults on data processing to the whole scientific community and user groups. Besides thepresentation of the IQmulus contest during the ISPRS Workshop Laser Scanning 2013,November11‐13,2013,weplantoproposethepresentationofthecontesttootherconferencesinthefield.

ThankstothefeedbackreceivedfromthemembersoftheAdvisoryBoardandtheparticipantsofthetracksofthefirstIQmuluscontest,newchallengesareplannedforthenexteditionsofthecontest,amongotherswelist:

ReconstructionfromLidarairbornescans(forinstance,D.LaeferwillmakeavailabletheLidarscansofthecenterofDublinobtainedintheERCgrantRETURN);

Featuredetection:extractionofsignificantshapecharacteristics; Changedetection,whichisanacuteissueingeospatialdataandrelatedtothetask4.5of

theIQmulusproject.

Moreover,forthenextyearsweplantostrengthenthelinkwiththepanelsofexpertsleavingthecoordinationofsometrackstomembersoftheAdvisoryBoard.Inaddition,weplantoincreasethenumberofmembersoftheAdvisoryBoard.

Finally,weplantosubmittheresultsofthecontestsforpublicationininternationallyrecognizedjournals.Amongotherswearecurrentlyconsideringjournalsintheremotesensingarea:

ISPRSJournalofPhotogrammetryandRemoteSensing,http://www.journals.elsevier.com/isprs‐journal‐of‐photogrammetry‐and‐remote‐sensing/

Remotesensing,http://www.mdpi.com/journal/remotesensing(openaccess) ComputersandGeosciences,

http://www.journals.elsevier.com/computers‐and‐geosciences/ IEEETransactionsonGeocienceandRemoteSensing

http://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=36

Inaddition,weareexploringtheorganizationofspecialissuesofjournalswithexecutablecode,following the example of the “Elsevier executable paper” based on the Collage AuthoringWorkbenchpublishedin2013bytheComputers&Graphicsjournal.

D8.8.1 IQmulus(FP7‐ICT‐2011‐318787)

15

8 REFERENCES

Berger, M., Levine, J., Nonato, L., Taubin, G. and Silva, C. T., 2013. A benchmark for surfacereconstruction.ACMTransactionsonGraphics32(2),pp.20:1–20:17.

Ciepiela,E.,Harezlak,D.,Kasztelnik,M.,Meizner,J.,Dyk,G.,Nowakowski,P.andBubak,M.,2013.TheCollageAuthoringEnvironment:FromProof‐of‐ConceptPrototypetoPilotService

Ciepiela, E., Nowakowski, P., Kocot, J., Harezlak, D., Gubala, T., Meizner, J., Kasztelnik, M.,Bartynski, T., Malawski, M. and Bubak, M., 2012. Managing entire lifecycles of e‐scienceapplications in the gridSpace2 virtual laboratory ‐ frommotivation through idea to operableweb‐accessibleenvironmentbuiltontopofPL‐gride‐infrastructure

Everingham,M.,VanGool,L.,Williams,C.K.I.,Winn,J.andZisserman,A.,2010.ThePascalVisualObjectClasses(VOC)Challenge.InternationalJournalofComputerVision88(2),pp.303–338.

Geiger,A.,Lenz,P.andUrtasun,R.,2012.ArewereadyforAutonomousDriving?TheKittiVisionBenchmark Suite. In: Conference on Computer Vision and Pattern Recognition (CVPR),Providence,RI,USA.

Geiger,A.,Lenz,P., Stiller,C. andUrtasun,R.,2013.VisionmeetsRobotics:TheKITTIDataset.InternationalJournalofRoboticsResearch(IJRR).

Gressin,A.,Mallet,C.,David,N.andDemantké,J.,2013.Towards3Dlidarpointcloudregistrationimprovement using optimal neighborhood knowledge. ISPRS Journal of Photogrammetry andRemoteSensing79,pp.240‐251.

Gressin,A.,Cannelle,B.,Mallet,C.andPapelard,J.‐P.,2012.Trajectory‐basedregistrationof3DlidarpointcloudsacquiredwithaMobileMappingSystems. ISPRSAnnalsofPhotogrammetry,RemoteSensingandtheSpatialInformationSciences,volI‐3.

Goyette,N.,Jodoin,P.‐M.,Porikli,F.,Konrad,J.,andIshwar,P.,2012.changedetection.net:Anewchangedetectionbenchmarkdataset.In:Proc.IEEEWorkshoponChangeDetection(CDW’12)atCVPR’12,Providence,RI,USA.

Nowakowski,P.,Ciepiela,E.,Harezlak,D.,Kocot,J.,Kasztelnik,M.,Bartynski,T.,Meizner,J.,Dyk,G.andMalawski,M.,2011.TheCollageAuthoringEnvironment

Monnier, F., Vallet, B., Paparoditis, N., Papelard, J.‐P. and David, N., 2013. Registration ofterrestrial mobile laser data on 2D or 3D geographic database by use of a non‐rigid ICPapproach. ISPRS Annals of Photogrammetry, Remote Sensing and the Spatial InformationSciences,volII‐W3.

Mueller,H.,Clough,P.,Deselaers,T.andCaputo,B.,2010.ImageCLEF:ExperimentalEvaluationinVisualInformationRetrieval.1stedn,SpringerPublishingCompany,Incorporated

Over, P., Awad, G., Michel, M., Fiscus, J., Sanders, G., Shaw, B., Kraaij, W., Smeaton, A. F. andQuenot,G.,2012.Trecvid2012–anoverviewof thegoals, tasks,data,evaluationmechanismsandmetrics.In:ProceedingsofTRECVID2012,NIST,USA.

Paparoditis, N., Papelard, J.‐P., Cannelle, B., Devaux, A., Soheilian, B., David, N. andHouzay, E.,2012. Stereopolis II: Amulti‐purpose andmulti‐sensor 3Dmobilemapping system for streetvisualisationand3Dmetrology.RevueFrançaisedePhotogrammétrieetdeTélédétection200:69‐79.

IQmulus(FP7‐ICT‐2011‐318787) D8.8.1

16

Pomerleau, F., Liu,M., Colas, F. and Siegwart, R., 2013. Challenging data sets for point cloudregistrationalgorithms.TheInternationalJournalofRoboticsResearch31(14),pp.1705‐1711.

Rottensteiner,F.,Sohn,G.,Jung,J.,Gerke,M.,Baillard,C.,Benitez,S.andBreitkopf,U.,2012.TheISPRS Benchmark on Urban Object Classification and 3D building. ISPRS Annals of thePhotogrammetry,RemoteSensingandSpatialInformationSciencesI‐3,pp.293–298.

Smeaton,A. F.,Over,P. andKraaij,W., 2006.Evaluation campaignsandTRECVid. In:MIR ’06:Proceedingsofthe8thACMInternationalWorkshoponMultimediaInformationRetrieval,ACMPress,NewYork,NY,USA,pp.321–330.

Smeaton, A. F., Over, P. and Kraaij, W., 2009. High‐Level Feature Detection from Video inTRECVid: a 5‐Year Retrospective of Achievements. In: A. Divakaran (ed.),Multimedia ContentAnalysis,TheoryandApplications,SpringerVerlag,Berlin,pp.151–174.

Strecha, C., von Hansen,W., Gool, L. V., Fua, P. and Thoennessen, U., 2008. On benchmarkingcamera calibration and multi‐view stereo for high resolution imagery. In: Conference onComputerVisionandPatternRecognition(CVPR),Anchorage,AK,USA.

Tombari,F.,Salti,S.andDiStefano,L.,2013.Performanceevaluationof3Dkeypointdetectors.InternationalJournalofComputerVision102(1‐3),pp.198–220.

Veltkamp,R.C.,Ruijsenaars,R.,Spagnuolo,M.,Zwol,R.andterHaar,F.,2006.SHREC2006:3Dshape retrieval contest. Technical Report UU‐CS‐2006‐030, Department of Information andComputingSciences,UtrechtUniversity.