Workshop Dubai Cut

8/19/2019 Workshop Dubai Cut

1/128

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Storage and Management of Large

Scale Point Cloud DataOracle Workshop9th International 3DGeoInfo 2014

Hans Viehmann1, Mike Horhammer1 ,Peter van Oosterom2 and Oscar Martinez Rubi3

1 Oracle 2 TU Delft 3 Netherlands eScience CenterNovember 11, 2014


2/128


Program Agenda

1

2

3

4

5

Introduction

File formats and tested datasets

Point cloud data management systems

Loading

Querying

Research and others

Conclusions

6

7


3/128


Program Agenda

1

2

3

4

5

Introduction



Loading

Querying

Research and others

Conclusions

6

7


4/128


Introduction2-year Netherlands eScience research project on Massive Point Clouds

• TU Delft:1. GIS technology

2. TU Delft Library

3. 3TU.Datacentrum

4. TU Delft Shared Service Center ICT

5. Cooperation with CWI/MonetDB

• NLeSC (Netherlands eScience Center)

• Oracle Corporation (NEDC)

• Rijkswaterstaat

• Fugro


5/128


Introduction

• Collection of point cloud data

grows rapidly – Many new applications now

economically viable

• Relevant use cases in GIS:

– Digital elevation model of terrains

– 3D models of urban environment

– Flood risk management

– Dike monitoring (NL)

– Forest mapping

• Requiring new techniqu

– Managing massive datas

– Integrating various kinds

• Other potential applica

– 3D models of manufactu

–

Visualization, animation, – Medical imaging

– Industrial metrology


6/128


IntroductionEvaluation of user requirements

•

Based on structured interviews with: – Government community: RWS (Ministry)

– Commercial community: Fugro (company)

– Scientific community: TU Delft Library

• Report at MPC public website http://

pointclouds.nl

• Basis for conceptual benchmark:

– Tests for functionality

– Classified by importance

• Testing single-user environments to avoid load-generator com

http://pointclouds.nl/http://pointclouds.nl/http://pointclouds.nl/


7/128


IntroductionPoint cloud data management/processing systems


8/128


Why use a database here?Requires a spatially enabled database

• Data integration with other sources – Online availability

– Geo-referenced imagery, existing 3D structures, attributes,...

• Fast access to arbitrary part of data set

– For processing or visualization

• General benefits of mature DBMS

– Information lifecycle management – data administration, tuning

– Scaleability – multi-user environment, multi-processor support, cluster

– Executing data-intense logic where the data resides


9/128Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Possible 3D processing flows in Oracle 12c

LiDAR Files

LiDAR loader

SDO_PC tables

or flat table

Query and Clip

Convert to

Geometries

Flat files

Load point

cloud

Point Tables

TGenerate TIN

Que

Co

Ge

G

Gen

Query and Clip

Calculate

Contour Lines


10/128Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

ÖBB-Infrastruktur AG, R&DOracle Spatial and Graph: Railway Network Management

Objectives

Optimize railway planning, construction andmaintenance

Enable efficient routing of railway lines

Solution

Stores and processes more than 8 billion

points of objects along railway tracks Enables LiDAR data to be viewed with existing

infrastructure vector data

Provides comprehensive metadata aboutrailway tracks through CSW

Delivers data through open WebServices

“[This technology…] is indispen

geospatial data with high efficicost,”

Dr. Michaela Haberler-WeberÖBB-Infrastruktur AG, R & D

Ima


11/128


Program Agenda

1

2

3

4

5

Introduction



Loading

Querying

Research and others

Conclusions

6

7


12/128


13/128


File formats and tested datasetsLAS: compression and in-file indexing

•Highly compressible

– LAZ (LAStools)

– zLAS (ESRI)

– Compression rate 10-15

• Indexable

– LAX (LAStools)

– LASX (ESRI)


14/128


File formats and tested datasetsActual Height Model of the Netherlands (AHN2)

•Covering surface of the entire country

• 6 -10 pts/m2 640 billion pts

• 60,185 LAZ files, 987 GB/11.64 TB

• (X, Y, Z) only

• “Future”

– AHN3 higher resolution

– Cyclorama-based photogrammetricdatasets (50x AHN2, and with RGB)


15/128


File formats and tested datasetsTested datasets

Name Points LAS files Disk size [GB] Area [km2] Description

20M 20,165,862 1 0.4 1.25 TU Delft campus

210M 210,631,597 16 4.0 11.25 Major part of Delft cit

2201M 2,201,135,689 153 42.0 125 City of Delft and surro

23090M 23,090,482,455 1,492 440.4 2,000 Major part of Zuid-Ho639478M 639,478,217,460 60,185 11,644.4 40,000 The Netherlands


16/128


File formats and tested datasetsBenchmarks

•“Mini-benchmark” – 1 dataset: 20M

– X, Y, Z

– Management systems:

• PointCloud (blocks) solutions in Oracle andPostgreSQL

• Flat tables in Oracle and PostgreSQL

• LAStools (not only X,Y,Z)

– 7 queries


17/128


18/128


File formats and tested datasetsBenchmarks queries


19/128


File formats and tested datasetsBenchmarks

•

Full benchmark – AHN2 (640bn points)

– X, Y, Z

– Management systems:

• Oracle EXADATA

•

Oracle blocks• PostgreSQL blocks

• LAStools

• MonetDB

– 30 queries


20/128


File formats and tested datasetsHardware & Software

•

Server HP Proliant DL380p Generation8 (2U rack) – 2 x 8-core Intel Xeon processors E5-2690, 2.9 GHz

– 128 GB RAM (DDR3)

– Disk storage:• 400 GB SATA SD device

• 300 GB SAS 10K rpm in RAID-1 configuration (internal, 2 disks)

• 6 TB SAS 15K rpm in RAID-5 configuration (internal, 22 disks)

• 2 x 41 TB SATA 7200 rpm in RAID-5 configuration (external in 4U rack 'Yotta-III' box, 24 disks)

– Network: 4 x 1 Gbit Ethernet

– Software:• Operating system: Red Hat Enterprise Linux, version 6.5

• DBMS: Oracle, PostgreSQL/PostGIS, MonetDB

• 'Geo' tools and libraries: CGAL, GDAL, GEOS, LAStools, PROJ.4, GeoTIFF, NetCDF, etc., etc.


21/128


File formats and tested datasetsHardware & Software

•

Oracle Exadata X4-2: Oracle SUN hardware for Oracle database software


22/128


Characteristics of Oracle Exadata Database Machin

Standard Database Servers

–

8x 2-socket servers

192 cores, 2TB DRAMor

– 2x 8-socket servers 160 cores, 4TB DRAM

Unified Ultra-Fast Network

– 40 Gb InfiniBand internal connectivity

– 10 Gb or 1 Gb Ethernet data center connectivity

Scale-out Intelligent Storage Servers

– 14x 2-socket servers 168 cores in storage

– 168 SAS disk drives 672 TB HC or 200 TB HP

– 56 Flash PCI cards 44 TB Flash + compressioFully Redundant


23/128


Oracle Exadata – Key innovations

Database offload in storage

–Data intensive query operations offloaded to

storage CPUs –100 GB/sec SQL data throughput

–Storage Index data skipping

Database optimized compression

–Hybrid Columnar for 10x DB size reductionand faster analytics

Database optimized PCI Flash

–Smart caching of database data

–2.66 Million Database IOs/sec

–Smart Flash log speeds transactions

Database optimize

–End-to-end prioritiza

application to DB and

Database optimize

–Fastest recovery of faserver, storage or sw

–Fastest backup. Incre

–Exachk top-to-bottom

hardware, software,

Database optimize

–SQL optimized InfiniBhigh throughput low


24/128


Oracle Exadata for Spatial Workloads

• Oracle Spatial and Graph is particularly suitable for the

Exadata architecture – CPU intensive geospatial calculations

– excellent use of parallelism

– load distribution between server nodes and storage heads

– index operations benefiting from parallelism and large memory

• Breaks new boundaries for ingesting spatial data – using the fast I/O and high network bandwidth

• Specific optimizations for point cloud data

– Excellent use of hybrid columnar compression (HCC)


25/128


Scalable from Eighth-Rack to Multi-Rack

Field Upgradeable - Supports

Multiple Generations of Hardware

Half Full

Multi-Rack

Eighth Quarter


26/128


Additional work packageStandardization

•

ISO/OGC spatial data: – at abstract/generic level, 2 types of spatial representations: features and coverages

– at next level (ADT level), 2 types: vector and raster, but perhaps points clouds should be

– at implementation/ encoding level, many different formats(for all three data types)

• nD point cloud:

– points in nD space and not per se limited to x,y,z(n ordinates of point which may also have m attributes)

– make fit in future ISO 19107 (as ISO 19107 is under revision).

– note: nD point clouds are very generic;e.g. also cover moving object point data: x,y,z,t (id) series.


27/128


Program Agenda

1

2

3

4

5

Introduction



Loading

Querying

Research and others

Conclusions

6

7


28/128


Point cloud data management systems (PC DMS)

• Oracle – Blocks

– Flat table

• PostgreSQL – Blocks

– Flat table

• LASTools

• MonetDB

Large scale point cloud data


29/128


PC DMS – Oracle Blocks


30/128


Point cloud data management systemsOracle blocks

•

Loading: – Global loading:

– Incremental loading:

• Blocking methods:

– R-Tree

– Hilbert R-Tree

•

Querying: – CLIP_PC function

– SDO_PC_NN (12.1.0.1)

– CLIP_PC_GET_MIN_HEIGHT (12.1

– CREATE_CONTOUR_GEOMETRIES


31/128


Point cloud data management systemsOracle blocks – Schema

Table Column:SDO_POINT_CLOUD

BLK_EXTENT:

SDO_GEOMETRY

BLK_IDOBJ_ID

TABLE of SDO_PC_B

• PC BLOB:

– n points

• (x, y, z, …): 8-byte IEEE doubles (


32/128


33/128


Point cloud data management systemsOracle blocks – Querying

Query


34/128


Point cloud data management systemsOracle blocks – Overlapping Blocks, Example Hilbert R-tree

Query


35/128


36/128


Point cloud data management systemsPostgreSQL blocks

•

Group points in blocks – Defined in 2D (x,y) space

– Block type: pcpatch

• PointCloud extension by P. Ramsey

https://github.com/pramsey/pointcloud

• Requires PostGIS

• Storage – Compression: 100 Mpts/GB

– Compression+: 200 Mpts/GB

•

Loading: – Incremental loading

• Require good input data spatial org

• Ordered loading (avoid de-clusterin

– External tool PDAL (pipeline)

http://www.pdal.io

– No parallel (doable with scripting,

• GIST (R-Tree) index for the blocks

• Querying:

– PC_Intersects and PC_Intersection

– No native parallel

https://github.com/pramsey/pointcloudhttps://github.com/pramsey/pointcloudhttp://www.pdal.io/http://www.pdal.io/https://github.com/pramsey/pointcloud


37/128


Point cloud data management systemsPostgreSQL blocks: PDAL chipper blocks

Images from http://www


38/128


PC DMS – LAStools


39/128


Point cloud data management systemsLAStools

• File-based solution by M. Isenburg

http://rapidlasso.com/lastools/

• Partly commercial:

– Some tools free open source

– Some tools require license (Windows)

– No license: noise, limit parallel

• Directly on LAS/LAZ

• Storage:

– LAS: 53 Mpts/GB

– LAZ: 540 Mpts/GB

• No data loading, but data prepar

– lastile, lassort, lasindex

• Require good input data spatial o

• Multi-core support (not really in

• Wine in Linux for Windows-only

• Hybrid solutions (DB) for massive

• Querying:

– lasmerge for rectangles and circle

– lasclip for polygons

– Polygon holes ignored

http://rapidlasso.com/lastools/http://rapidlasso.com/lastools/http://rapidlasso.com/lastools/


40/128


PC DMS – Oracle Flat


41/128


Point cloud data management systemsOracle flat

• Pro

– No blocking required fast creation

– Dynamic updates fast updates

– Flexible schema

• Con

– Queries less scalable on non-Exadata(< 1 billion pts)

• Storage: 23 Mpts/GB (200 Mpts/GB on Exadata)

• Indexing

– None on Exadata

– B-tree on (x, y)

– IOT works well

• Querying:

– Plain old SQL

• Range query for rectangles

• Distance method for circles

– SDO_PointInPolygon for polygons


42/128


PC DMS – PostgreSQL Flat


43/128


Point cloud data management systemsPostgreSQL flat

• Points in flat table

– Row per point

– Data type: Double precission

• Storage: 13 Mpts/GB

• Loading:

– Las2txt + “COPY FROM”

– Faster NLeSC binary loader (convert LAS to PostgreSQLbinary dump format)

– No parallel (doable with scripting, e.g. Python)

• B-Tree index on X,Y

– GIST too expensive

– No parallel

• Querying:

– No native parallel

– Range query for rectangles

–

Distance method for circles – _ST_Contains,ST_MakePoint for p

• Bad scaling: Only use for small da


44/128


PC DMS – MonetDB


45/128


Point cloud data management systemsMonetDB

• Column store

• Points in flat table

– Row per point

– Data type: Double precission

• Storage: 40 Mpts/GB

• Loading:

– Las2txt + “COPY FROM”

– Faster NLeSC binary loader (convert LAS to MonetDBcolumnar format)

– Parallel

• Imprints indexing

• Querying:

– Parallel

• Active development

• Not released yet

• Bad scaling: Only use for small da


46/128


Point cloud data management systemsStorage models

Storage model Pro ConDB blocks • Storage (compression)

• Scaling• Indexing• DB functionalities• Complex queries

• Loading (make blocks)• Block overhead in queries(noticeable in simple queries)

DB flat • Faster loading/updating• DB functionalities• Dynamic schema• Simple queries

• Storage (except Exadata)• Not scaling (except Exadata)• Indexing (except Exadata)

File-based • Storage (LAZ)• Data preparation• Simple queries (if not LAZ)

• Limited functionalities• Fixed schema (LAS)• Scaling requires DB help


47/128


Program Agenda

1

2

3

4

5

Introduction



Loading

Querying

Research and others

Conclusions

6

7


48/128


Loading – Oracle Blocks


49/128


LoadingOracle Blocks – Loading Process

LAZ

LAS

CSV

Flat table

SDO_PC

LASzip CREATE_PC (1

SqlLoader & preprocessor (4)


50/128



LAZ

LAS

CSV

Flat table

SDO_PC

LASzip CREATE_PC

SqlLoader & preprocessor


51/128


LoadingOracle Blocks (Hilbert blocking) – Create tables

CREATE TABLE AHN_BLCK

TABLESPACE USERSPCTFREE 0NOLOGGINGLOB(POINTS)STORE AS SECUREFILE (TABLESPACE USERS COMPRESS HIGH) AS

SELECT * FROM MDSYS.SDO_PC_BLK_TABLE WHERE 0 = 1;

CREATE TABLE AHN_BASE (id number,PC SDO_PC)

TABLESPACE USERS

PCTFREE 0NOLOGGING;


52/128


LoadingOracle Blocks (Hilbert blocking) – Create PC

declare

pc sdo_pc;beginpc :=

sdo_pc_pkg.init(basetable => 'PCS',basecol => 'PC',blktable => 'BLOCKS',ptn_params => 'blk_capacity=10000',pc_extent => mdsys.sdo_geometry(2003, null, null,

mdsys.sdo_elem_info_array(1, 1003, 3),mdsys.sdo_ordinate_array(289020.90, 4320942.61,

290106.02, 4323641.57)),pc_tol => 0.05,

pc_tot_dimensions => 3,pc_domain => null,pc_val_attr_tables => null,pc_other_attrs => xmltype(

' '));

sdo_pc_pk

inpinptabclstpc

insert i

end;

/


53/128


LoadingOracle Blocks (Hilbert blocking) – Metadata, Index

•

Update the SRID of the blocks, drop the staging table and add the blocks UPDATE AHN_BLCK B SET B.BLK_EXTENT.SDO_SRID = 28992;

DROP TABLE AHN_STAGING;

INSERT INTO USER_SDO_GEOM_METADATA VALUES (’AHN_BLCK’,’BLK_EXTENT’,SDO_DIM_ARRAY(SDO_DIM_ELEMENT(’X’, [min. X], [max. X],0.0001),SDO_DIM_ELEMENT(’Y’, [min. Y], [max. Y],0.0001)),

28992);

CREATE INDEX AHN_BLCK_SIDXON AHN_BLCK (BLK_EXTENT)INDEXTYPE IS MDSYS.SPATIAL_INDEXPARAMETERS (’TABLESPACE=INDX WORK_TABLESPACE=PCWORKLAYER_GTYPE=POLYGON SDO_INDX_DI

SDO_RTR_PCTFREE=0’)PARALLEL 16;


54/128



LAZ

LAS

CSV

Flat table

SDO_PC

LASzip CREATE_PC



55/128


LoadingOracle Blocks (Hilbert blocking) – Incremental Loading

$JDKHOME/bin/java -Xms2048m -classpath /…/ojdbc6.jar:/…/sdoutl.jar oracle.spatial.util.Las2SqlLdrIndep 1 BLOCKS BLOCK_ID_/…/ahn_bench023090_01.las 10000 jdbc:oracle:thin:@//host:port/sid largepc largepc 100000000 (10K block capaciConnecting to jdbc:oracle:thin:@//host:port/sid...Reading 100,000,000 points... 23.324 s. Sorting: 65.362 s. Writing: 164.608 sReading 99,995,084 points... 28.793 s. Sorting: 63.231 s. Writing: 187.621 sReading 99,997,274 points... 38.275 s. Sorting: 138.074 s. Writing: 216.841 sReading 99,991,996 points... 50.051 s. Sorting: 141.184 s. Writing: 236.634 sReading 99,990,636 points... 54.492 s. Sorting: 161.936 s. Writing: 240.436 sReading 99,990,636 points... 58.812 s. Sorting: 149.797 s. Writing: 244.515 sReading 99,990,637 points... 58.353 s. Sorting: 148.561 s. Writing: 242.785 sReading 99,990,638 points... 60.001 s. Sorting: 151.770 s. Writing: 242.187 sReading 99,990,639 points... 61.795 s. Sorting: 153.668 s. Writing: 243.461 sReading 99,990,640 points... 60.733 s. Sorting: 150.150 s. Writing: 243.077 sReading 99,990,641 points... 60.419 s. Sorting: 151.012 s. Writing: 244.065 sReading 99,999,834 points... 62.434 s. Sorting: 151.908 s. Writing: 246.490 sReading 99,990,124 points... 58.342 s. Sorting: 152.249 s. Writing: 245.693 sReading 99,999,835 points... 57.816 s. Sorting: 151.361 s. Writing: 244.788 s

Reading 99,990,145 points... 56.882 s. Sorting: 153.129 s. Writing: 245.248 sReading 99,997,246 points... 56.489 s. Sorting: 152.962 s. Writing: 244.226 sReading 99,992,320 points... 58.604 s. Sorting: 150.968 s. Writing: 253.848 sReading 99,990,606 points... 59.011 s. Sorting: 162.808 s. Writing: 253.697 sReading 99,990,607 points... 56.012 s. Sorting: 151.588 s. Writing: 255.625 sReading 99,990,608 points... 57.417 s. Sorting: 152.561 s. Writing: 256.142 sReading 139,854 points... 0.110 s. Sorting: 0.097 s. Writing: 0.375 s------------------------------------------------------------------------------------------Read 2,000,000,000 points... 1,078.165 s. Sorting: 2,854.376 s. Writing: 4,752.362 s (~⅔ sequentialTime elapsed......................... 8,708.131 s


56/128


Loading – PostgreSQL Blocks


57/128


LoadingPostgreSQL blocks

• Load extensions and create blocks table (SQL):CREATE EXTENSION postgis;

CREATE EXTENSION pointcloud;

CREATE EXTENSION pointcloud_postgis;

CREATE TABLE patches (id SERIAL PRIMARY KEY, pa PCPATCH);

• Requires XML with data format inserted in pointcloud_formats

– Attributes

–

Scales and offsets – Compression: “none” or dimensional

– Multiple formats possible

INSERT INTO pointcloud_formats (pciSRID ,’


58/128


LoadingPostgreSQL blocks

• PDAL (Command-line) with configuration XML:

– pcid as in pointclouds_format

– Block size: Chipper filter capacity (smaller than Oracle)

– Selector filter to select attributes

– Multiple PDAL processes possible

pdal pipeline [xml File]

• Create a PostGIS GIST index (SQL):CREATE INDEX pa_gix ON blocks USING GIST(geometry(pa));

• VACUUM and ANALYZE (SQL)

– optional

VACUUM FULL ANALYZE blocks;

host=’dbhost’ dbn

user=’dbuser ’ blocksSRID false1

BlockSize


59/128


Loading – LAStools


60/128


LoadingLAStools

• Optional tiling (if bad spatial organization)lastile –i *.las –o tile.las

• Resort and index (Command-line):

lassort.exe *las

lasindex *las

• Multi-core:

– option “cores”

lassort.exe *las –cores 8

lasindex *las –cores 8

– Scripting

lassort.exe -i [input LASfile] -o [output LASfile]

lasindex -i [output LASfile]


61/128


LoadingLAStools (Hybrid: PostgreSQL-PostGIS)

• Create DB and table for LAS files extents (SQL).CREATE EXTENSION postgis;CREATE TABLE lasindex (filepath text, geom public.geometry(Geometry,28992));

• Extract header information (lasinfo) and insert in DB:

– Multiple processes possiblelasinfo [LAS file] -nc -nv -ncoINSERT INTO lasindex (filepath,geom) VALUES (’[LAS file]’, ST_MakeEnvelope(minx ,miny ,

• Create the GIST index on the extents (SQL).

CREATE INDEX ON lasindex USING GIST (geom);

• VACUUM and ANALYZE (SQL)

– optional

VACUUM FULL ANALYZE lasindex;


62/128


Loading – Oracle Flat (Exadata)


63/128


LoadingOracle flat (Exadata) – Loading Process

LAZ

LAS

CSV

Flat table

SDO_PC

LASzip CREATE_PC



64/128


LoadingOracle flat (Exadata) – SqlLoader from CSV / ASCIIconnect sys as sysdbacreate or replace directory las_dir as

'/scratch/…';

CREATE TABLE lidar_ext (val_d1 NUMBER,val_d2 NUMBER,val_d3 NUMBER,d NUMBER)

ORGANIZATION EXTERNAL (TYPE ORACLE_LOADERDEFAULT DIRECTORY las_dirACCESS PARAMETERS (RECORDS DELIMITED BY NEWLINEFIELDS (

val_d1 CHAR(8),val_d2 CHAR(8),val_d3 CHAR(8),d CHAR(8)))

LOCATION ('lidar1.txt','lidar2.txt','lidar3.txt'))

PARALLEL;

create table lidar as select * fromlidar_ext;

more lidar*txt::::::::::::::

lidar1.txt::::::::::::::

1 1 1 123

2 2 2 124

3 3 3 125::::::::::::::

lidar2.txt

::::::::::::::

4 4 4 1235 5 5 124

6 6 6 125

::::::::::::::

lidar3.txt::::::::::::::

7 7 7 1238 8 8 124

9 9 9 125


65/128


LoadingOracle flat (Exadata) – Loading Process

LAZ

LAS

CSV

Flat table

SDO_PC

LASzip CREATE_PC



66/128


LoadingOracle flat (Exadata) – SqlLoader from LAScreate directory las_dir as ‘…';create directory exe_dir as ‘…';create directory log_dir as ‘…';

grant read on directory las_dir to mdsys;grant read, execute on directory exe_dir to

mdsys;grant write on directory log_dir to mdsys;

create table inptab_ext(val_d1 number,val_d2 number,val_d3 number,d number)

organization external (type oracle_loaderdefault directory las_dir

access parameters (records delimited by newline preprocessor exe_dir:'preprocessor.sh'BADFILE log_dir:'preprocess.bad'LOGFILE log_dir:'preprocess.log'fields terminated by ",")

location ('Serpent_Mound_Model_LAS_Data.las','Serpent_Mound_Model_LAS_Data2.las','Serpent_Mound_Model_LAS_Data3.las','Serpent_Mound_Model_LAS_Data4.las'))

parallel 16reject limit unlimited;

preprocessor.sh

#!/bin/bash

$JDKHOME/bin/java -classpath /scratch/mhorhamm/sdoutl.jaroracle.spatial.util.Las2SqlLdr $1


67/128


LoadingOracle flat (Exadata) – Materialize table & Create Index (no for Exadata)CREATE TABLE AHN_FLATTABLESPACE USERSPCTFREE 0

NOLOGGINGPARALLEL 16 ASSELECTVAL_D1,VAL_D2,VAL_D3

FROMINPTAB_EXT;

CREATE INDEX AHN_FLAT_IDXON AHN_FLAT (VAL_D1,

VAL_D2)TABLESPACE INDXPCTFREE 0NOLOGGINGPARALLEL 16;


68/128


LoadingTuning

• Tune DB server parameters (working memory, …)

• Use different file-system for DB and input data (faster IO)

• If available use SSD disks for indexes (different table space)

• > 40bn points consider partitions

• Block sizes Oracle > PostgreSQL (also depend on size typical query regions)

• Input data not spatial overlap Incremental loading

• Compression: Better storage, slower queries and loading

• Not native parallel easy do-it-your-own (be careful with de-clustering)


69/128


LoadingMedium benchmark comparison

PostgreSQL flat: Binary loader, not native parallel16 by script, index not parallel

Oracle flat: SQL loader from LAS (ext. table, pre-processor script) , parallel 16

MonetDB flat: Binary loader, parallel

PostgreSQL blocks: Incremental loading with PDAL,no native parallel 12 by script, blocksize 3000

Oracle blocks: Global loading, SQL loader from LAS,Hilbert blocking, parallel 16, blocksize 10000

LAStools: (unlicensed), LAS files, no native parallel16 by script , hybrid DB solution (PostgreSQL)

Oracle EXADATA X4-2 half rack: 96 cores, 5compression levels, SQL loader from LAS, flat table,no index


70/128


71/128


LoadingMedium benchmark comparison: Loading (inc. indexing) of 23090M point

0,33 0,12

1,65

0,51

3,302,54

1,46

0,24

2,72

14,09

0,00

5,00

10,00

15,00

20,00

25,00

Oracle blocksglobal

Oracle blocksglobal compress

Oracle blocksincrem.

Oracle blocksincrem. compress

PostgreSQL blocksincrem.

PostgreSQL blocksincrem. compress

Oracle flat PostgreSQL flat MonetDB LAStools

Loading [Mpts/s]


72/128


LoadingMedium benchmark comparison: Loading (inc. indexing) of 23090M point

0,33

0,12

1,65

0,51

3,30

2,54

1,46

0,24

2,72

0,00

0,50

1,00

1,50

2,00

2,50

3,00

3,50

4,00

Oracle blocksglobal







Loading [Mpts/s]


73/128


LoadingMedium benchmark comparison: Storage of 23090M points

32,99

104,91

32,99

104,91 115,45

216,20

23,16 12,97

43,62 52,42

0,00

100,00

200,00

300,00

400,00

500,00

600,00

700,00

Oracle blocksglobal







Storage [Mpts/GB]


74/128


Program Agenda

1

2

3

4

5

Introduction



Loading

Querying

Research and others

Conclusions

6

7


75/128


Querying – Oracle Blocks


76/128


QueryingOracle blocks – CLIP_PC (return blocks)

SQL> select

blk_id,num_pointsfromtable(sdo_pc_pkg.clip_pc(inp => (select pc from pcs where id = 1),ind_dim_qry => sdo_geometry(

2003,null,null,sdo_elem_info_array(1, 1003, 3),

sdo_ordinate_array(10, 10, 14, 14)),other_dim_qry => null,qry_min_res => 1,qry_max_res => 1));

BLK_ID NUM_POINTS ---------- ----------

53 25


77/128


QueryingOracle blocks – CLIP_PC (return individual point records)

select

query_points.x,query_points.y,query_points.z

fromtable(sdo_pc_pkg.clip_pc(inp => (select pc from pcs where id = 1),ind_dim_qry => sdo_geometry(

2003, null, null,sdo_elem_info_array(1, 1003, 3),sdo_ordinate_array(10, 10, 14, 14)),

other_dim_qry => null,qry_min_res => 1,qry_max_res => 1)) query_blocks,

table(sdo_util.getvertices(geometry => sdo_pc_pkg.to_geometry(

pts => query_blocks.points,num_pts => query_blocks.num_points,

pc_tot_dim => 3,srid => null))) query_points;


78/128


QueryingOracle blocks – CLIP_PC (query by non-indexed dim)

selectquery_points.x,

query_points.y,query_points.z,query_points.w Intensity -- followed by v5 - v11

fromtable(sdo_pc_pkg.clip_pc(inp => (select pc from pcs where id = 1),ind_dim_qry => sdo_geometry(

2003, null, null,sdo_elem_info_array(1, 1003, 3),sdo_ordinate_array(10, 10, 14, 14)),

other_dim_qry => sdo_mbr(lower_left => sdo_vpoint_type(123, 1),upper_right => sdo_vpoint_type(123, 1000)),

qry_min_res => 1,

qry_max_res => 1)) query_blocks,table(sdo_util.getvertices(

geometry => sdo_pc_pkg.to_geometry( pts => query_blocks.points,num_pts => query_blocks.num_points,

pc_tot_dim => 4,srid => null))) query_points;

X Y Z INTENSITY---------- ---------- ---------- ----------

10 13 123 456…


79/128


QueryingOracle blocks – CLIP_PC (support non-blocked dimensions)selectquery_points.x,query_points.y,query_points.z,query_points.w Intensity, -- followed by v5 - v11out.non_blocked_dim

fromtable(

sdo_pc_pkg.clip_pc(inp => (select pc from pcs where id = 1),ind_dim_qry => sdo_geometry(2003, null, null, sdo_elem_info_array(1, 1003, 3), sdo_ordinate_array(10, 10, 14, 14)other_dim_qry => sdo_mbr(

lower_left => sdo_vpoint_type(123, 1),upper_right => sdo_vpoint_type(123, 1000)),

qry_min_res => 1,qry_max_res => 1)) query_blocks,

table(sdo_util.getvertices(geometry => sdo_pc_pkg.to_geometry(

pts => query_blocks.points,num_pts => query_blocks.num_points, pc_tot_dim => 4,srid => null ,get_ids => 1))) query_points ,

pcs_out out where

out.ptn_id = query_points.v5 and -- v5 is blk_id, v6 is pt_id out.point_id = query_points.v6;

X Y Z INTENSITY NON_BLOCKED_DIM---------- ---------- ---------- ---------- --------------------------------

11 10 123 456 # 21…


80/128


QueryingOracle blocks – SDO_PC_NN

SQL> select2 rownum pt_pos,3 sdo_geometry(4 3001,

5 null,6 sdo_point_type(x, y, z),10 null,11 null) pts12 from13 table(14 sdo_util.getvertices(15 geometry => sdo_pc_pkg.to_geometry(16 pts => sdo_pc_pkg.sdo_pc_nn(17 pc => (select pc from pcs where pc_id = 1),18 center => sdo_geometry(3001, null, sdo_point_type(15, 15, 30), null, null),24 n => 3200),25 num_pts => 3200,26 pc_tot_dim => 3,

27 srid => null,28 blk_domain => null,29 get_ids => 1)));

PT_POS PTS(SDO_GTYPE, SDO_SRID, SDO_POINT(X, Y, Z), SDO_ELEM_INFO, SDO_ORDINATES)------------------------------------------------------------------------------------------1 SDO_GEOMETRY(3001, NULL, SDO_POINT_TYPE(15, 15, 30), NULL, NULL)2 SDO_GEOMETRY(3001, NULL, SDO_POINT_TYPE(15, 14, 29), NULL, NULL)

…3200 SDO_GEOMETRY(3001, NULL, SDO_POINT_TYPE(43, 44, 87), NULL, NULL)


81/128


Querying – PostgreSQL Blocks


82/128


QueryingPostgreSQL blocks – PC_Intersects / PC_Intersection (return blocks)

SELECT

PC_Intersection(pa,geom)

FROM patches, query_polygons

WHERE PC_Intersects(pa,geom) and query_polygons.id = 1;

CREATE TABLE query_polygons (id integer, geom public.geometry(Geometry, SRI

INSERT INTO query_polygons VALUES (1,ST_GeomFromEWKT(WKT));


83/128


QueryingPostgreSQL blocks – PC_Intersects / PC_Intersection (return individual po

SELECT

PC_Get(qpoint, 'x') AS x,PC_Get(qpoint, 'y') AS y, PC_Get(qpoint, 'z') A

FROM

(

SELECT

PC_Explode(PC_Intersection(pa,geom)) AS qpoint

FROM patches, query_polygons

WHERE PC_Intersects(pa,geom) and query_polygons.id = 1

) AS qtable;


84/128


QueryingPostgreSQL blocks – Nearest Neighbor

SELECTPC_Get(qpoint, 'x') as x, PC_Get(qpoint, 'y') as y, PC_Get(qpoint, 'z') a

FROM(SELECT

PC_explode(pa) as qpointFROM

(SELECTpa

FROM patches ORDER BY geometry(pa) ST_SetSRID(ST_MakePoint( px , py ),LIMIT 9

) as A) as BORDER BY ((PC_Get(qpoint, 'x') - px )^2 +(PC_Get(qpoint, 'y') - py )^2)LIMIT 1000;


85/128


Querying – LAStools


86/128


QueryingLAStools – lasmerge

• If hybrid approach with PostgreSQL-PostGIS:

CREATE TABLE query_polygons (id integer, geom public.geometry(Geometry, S

INSERT INTO query_polygons VALUES (1,ST_GeomFromEWKT(WKT));

psql dbname -U dbuser -h dbhost -p dbport -t -A -c \

“SELECT l.filepath FROM lasindex l,query_polygons qp WHERE ST_Intersects(qp.geom, l.\

> input1.list

• Rectangle:lasmerge -lof input.list -inside minx miny max x max y -o output.las

• Circle:

lasmerge -lof input.list -inside_circle cx cy r -o output.las


87/128


QueryingLAStools – lasclip (query region in ShapeFile)

pgsql2shp -f query5.shp -h dbhost -p dbport -u dbuser -P dbpass dbname \

“select ST_SetSRID(geom, 28992) from query_polygons where id = 5;”

lasclip.exe -lof input5.list -poly query5.shp -o output5.las -merged

• If hybrid approach with PostgreSQL-PostGIS:

psql dbname -U dbuser -h dbhost -p dbport -t -A -c \

“SELECT l.filepath FROM lasindex l,query_polygons qp WHERE

ST_Intersects(qp.geom, l.geom) AND qp.id=5" \

> input5.list


88/128


QueryingLAStools

• Tune LAZ chunksize (default 50000) depending on your type of queries

• lassort of large files done with sub-tiling

• lasclip/lasmerge option io_buffer (default 262144 bytes) for granular accesses

• Ask in LAStools forum


89/128


Querying – Oracle Flat


90/128


91/128


QueryingOracle flat – SDO_PointInPolygon

• Other regions (example circle):SQL> SELECTx,y,zFROMTABLE(mdsys.sdo_PointInPolygon(CURSOR(

selectval_d1 x,val_d2 y,val_d3 z

fromflat),

MDSYS.SDO_GEOMETRY(2003, NULL, NULL,MDSYS.SDO_ELEM_INFO_ARRAY(1, 1003,4), -- circle with

MBR(10,10, 14,14)MDSYS.SDO_ORDINATE_ARRAY(10, 12, 12, 10, 14, 12)),0.05));

X Y Z---------- ---------- ----------

10 12 123…

13 rows selected.Elapsed: 00:00:13.16

• Other regions (with pre-filtering):SQL> SELECTx, y, zFROM

TABLE(mdsys.sdo_PointInPolygon(CURSOR(selectval_d1 x,val_d2 y,val_d3 z

fromflat

whereval_d1 between 10 and 14 and

val_d2 between 10 and 14),MDSYS.SDO_GEOMETRY(2003, NULL, NULL,MDSYS.SDO_ELEM_INFO_ARRAY(1, 1003, 4

(10, 10, 14, 14)MDSYS.SDO_ORDINATE_ARRAY(10, 12, 12

X Y Z---------- ---------- ----------

10 12 123…

13 rows selected.Elapsed: 00:00:00.00


92/128


QueryingOracle flat

• Nearest neighbor - Approximation:

SELECT*

FROM(SELECT*

FROMflat

WHERE((x BETWEEN (px – r) AND (px + r)) AND(y BETWEEN (py – r) AND (py + r))

)ORDER BYPOWER(x - px, 2) + POWER(y - py, 2)

) aWHEREROWNUM


93/128


QueryingMedium benchmark comparison

PostgreSQL flat: parallel

querying with PQGR algorithm

Oracle flat: native parallel

MonetDB flat: native parallel

PostgreSQL blocks: : parallelquerying with PQGR algorithm

Oracle blocks: : parallelquerying with PQGR algorithm

LAStools: single-process

Oracle EXADATA X4-2 half rack:only 23090M


94/128


QueryingMedium benchmark comparison: 20M and 23090M; Queries #1 and #6

Query 1

20M

Query 1

23090M

Query 6

20M

Query 6

23090MOracle blocks global* 0,35 0,35 1,20 1,20

Oracle blocks global compress* 0,41 0,41 1,38 1,40

Oracle blocks increm.* 0,35 0,35 1,20 1,20

Oracle blocks increm. compress* 0,41 0,41 1,38 1,40PostgreSQL blocks increm.* 0,25 0,25 1,50 1,50PostgreSQL blocks increm. compress* 0,32 0,32 1,72 1,68

Oracle flat 0,24 1,25 1,32 27,18

PostgreSQL flat* 0,35 5,17 1,72 15,46

MonetDB 0,06 7,21 187,16 198,51

LAStools** 0,02 0,03 1,46 1,48

LAStools LAZ** 0,14 0,12 1,90 1,91

Oracle Exadata 0,18 0,67


95/128


QueryingMedium benchmark comparison: Queries #1 Rectangle 51x53 m, 0.0027 k

0,00

0,05

0,10

0,15

0,20

0,25

0,30

0,35

0,40

0,45

0,50

Oracle blocksglobal




PostgreSQLblocks increm.

PostgreSQLblocks increm.

compress


T i m e [ s ]

Query #1 response time


96/128


97/128


Program Agenda

1

2

3

4

5

Introduction



Loading

Querying

Research and others

Conclusions

6

7


98/128


Create PC with pyramiddeclare

pc sdo_pc;begin

pc :=sdo_pc_pkg.init(basetable => 'PCS', --(base table)basecol => 'PC', --(base column)blktable => 'BLOCKS', --(data table)ptn_params => 'blk_capacity=10000', --(block capacitpc_extent => mdsys.sdo_geometry( --(spatial exten

2003,null,null,mdsys.sdo_elem_info_array(1, 1003, 3),mdsys.sdo_ordinate_array(1, 1, power(2, 13), power(2, 13))),

pc_tol => 0.5, --(tolerance)pc_tot_dimensions => 3, --(total dimensipc_domain => null,pc_val_attr_tables => null,pc_other_attrs => xmltype( --(object metada

'


99/128


Query with LoDselect

query_points.x,query_points.y,query_points.z

fromtable(sdo_pc_pkg.clip_pc(inp => (select pc from pcind_dim_qry => sdo_geometry(

2003,null,null,sdo_elem_info_arsdo_ordinate_arr

other_dim_qry => null,qry_min_res => 1,qry_max_res => 1)) query_blocks,

table(sdo_util.getvertices(

geometry => sdo_pc_pkg.to_geometry( pts => query_bnum_pts => query_b

pc_tot_dim => 3,srid => null)))

Research and othersLevel of detail

• Currently available in OracleSDO_PCR-tree, SDO_TIN, not yetSDO_PCHilbert – Thinning of points

• Currently single simple approach

• Others planned, maybe based on slope, category,intensity, ...

• Maybe customizable with Java function...

1

2

1

3

1

2

1

4

1

2

1

3

1

2

1 1

2

1

3

1 2 1 3 1


100/128


Research and othersLevel of detail


101/128


Research and othersParallel querying algorithms

• Parallel Querying with Candidate blocks (PQCB)

• Parallel Querying with Gridded Region (PQGR)


102/128


103/128


Research and othersParallel Querying with Gridded Region: Example PostgreSQL blocks (Pytho

-- Create table with 8 sub-regions of query region

CREATE TABLE query_grid_1 AS

WITH A AS (

SELECT geom, st_xmin(geom) as minx, st_xmax(geom) as maxx,

st_ymin(geom) as miny, st_ymax(geom) as maxy

FROM query_polygons where id = 1)

SELECT B.row, B.col,

ST_Intersection(A.geom, ST_SetSRID(B.geom, ’28992’)) as geom

FROM

A,

(SELECT F.row, F.col, F.geom

FROM A,

ST_CreateFishnet(2, 4, (maxx-minx) / 4,

(maxy-miny) / 2, minx, miny) F

) AS B ;

CREATE INDEX query_grid_1_rowcol ON query_grid_1 ( row, col );

-- Create table to store the results

CREATE TABLE query_results_1 (x NUMERIC, y NUMERIC, z NUMERIC);

-- The 8 processes insert data in parallel (example for pr

INSERT INTO

query_results_1 ( x,y,z )

SELECT PC_Get(qpoint, ’x’) as x,

PC_Get(qpoint, ’y’) as y,

PC_Get(qpoint, ’z’) as z

FROM

(SELECT PC_explode(pc_intersection(pa,geom)) a

FROM patches, query_grid_1

WHERE pc_intersects(pa,geom) AND row = 1 AN

-- Remove possible duplicated points

CREATE TABLE cleaned_query_results_1 AS SELECT DISTI

http://trac.osgeo.org/postgis/

h d h


104/128


Research and othersFaster flat tables: Query comparison

0

1

2

3

4

5

6

7

8

20M 210M 2201M 23090M

T i m e [ s ]

Query #1 (rectangle 51 x 53 m, ~74872 points)

Po

O

M

Po

O

LA

h d h


105/128


Research and othersFaster flat tables

• Flat tables fast when small dataset but not scaling

• Blocks not so fast but “perfectly” scaling (better storage but need “blocks” processing in

• What if we want DBMS with speed of flat tables but good scaling? (we do not care abou

h d h


106/128


Research and othersFaster flat tables: spatial clustering

• Reorder data based on spatial dimensions

– Efficiency in storage and queries

– Already used in:

• LAStools (lassort)

• Oracle blocks (Data preparation in Hilbert R-Tree blocking)

• Space filling curves: Hilbert/Morton

– Useful for flat model directly

–X, Y -> Code (position in the curve) No need to store X,Y!

– Can also be used in cases where point dimension is not per se spatial (x, y, z), but of different nLoD/imp)

– Queries need to be re-written to use spatial clustering and be more efficient

R h d h


107/128


Research and othersFaster flat tables: Current approach flat table

0 1 2 3 y

x

3

2

1

0

x y pos

0 0 0

0 1 2

0 2 3

0 3 9

1 0 1

1 1 4

…

index

0

15

Example input data

(sorted as acquired)

x y

0 0

1 0

0 1

0 2

1 1

....

Flat table B-Tree XY

R h d h


108/128


Research and othersFaster flat tables: Current approach flat table with clustering (on B-Tree)

0 1 2 3 y

x

3

2

1

0

0

15x y pos

0 0 0

0 1 1

0 2 2

0 3 3

1 0 4

1 1 5

…

clustering

0 1 2 3 y

x

3

2

1

0

x y pos

0 0 0

0 1 2

0 2 3

0 3 9

1 0 1

1 1 4

…

0

15 x y

0 0

1 0

0 1

0 2

1 1

....

Flat table B-Tree XY Flat table

R h d th


109/128


Research and othersFaster flat tables: Space filling curves

0 1 2 3 y

x

3

2

1

0

0

15

0 1 2 3 y

x

3

2

1

0

0

15

0 1 2 3 y

0 15

x

3

2

1

0

(first y, then x) Morton / PeanoHilbert

Bad spatial clustering Better spatial clustering

R h d th


110/128


Research and othersFaster flat tables: Constructing Morton curve

0

00 01

row

01

00

3

0 1 2 3 col

row

3

2

1

0

0

15

Peano or Morton or N-order (or Z-order)

Recursively replace each vertex of basic curvewith the previous order curve

Alternative: bitwise interleaving

Also works in 3D (or nD)

0 1 2 3

3

2

1

0

0

4 5 6

row

7

6

5

4

R h d th


111/128


Research and othersFaster flat tables: 3D Morton curve

Illustration

2x2x2 4x4x4

R h d th


112/128


Research and othersFaster flat tables: Constructing Hilbert curve

Hilbert

rotate previous order curveat vertex 0 (-) and vertex 3 (+)

Also in 3D

00 01

row

01

000 3

0 1 2 3 col

row

3

2

1

0

0 150 6

R h d th


113/128


Research and othersFaster flat tables: 3D Hilbert curve

Illustrations

2x2x2 4x4x4 8x

R h d th


114/128


Research and othersFaster flat tables: Average number of clusters for all possible range querie

N=8, number of Clusters for a given range query:

0 63Peano (3 ranges) Hilbert (2 ranges)

N*N

GRID

HILBERT PEAN

2*2 1.11 1.22

4*4 1.64 2.16

8*8 2.93 4.41

16*16 5.60 9.29

Faloutsos and Ros

Research and others


115/128



• LAStools lassort+lasindex

• Oracle Hilbert blocking

• PostgreSQL Morton/QuadTree blocking (NLeSC/TU Delft test)

• SFC flat table model (NLeSC/TU Delft test)

– compute Hilbert / Morton codes for all points

– create b-tree index on Hilbert / Morton code (position in the curve)

– cluster flat table on Hilbert / Morton curve

– re-write queries to select ranges of codes

Research and others


116/128



• Better SFC flat table model:

– Not use the default heap-table, but an indexed organized table (not PostgreSQL)

– No separate index structure needed -> more compact, faster (not PostgreSQL)

– No need to store encoded dimensions (if 2D: x, y not necessary -> only code, z)

Research and others


117/128


Research and othersFaster flat tables: SQL DDL (2D morton codes, still using X, Y)

• Oracle

• PostgreSQL

• MonetDB

CREATE TABLE flat (x,y,z,morton);

[… BINARY LOAD …]

CREATE INDEX morton_index ON flat (morton);

CLUSTER flat using morton_index;

CREATE TABLE flat_temp (x,y,z,morton);[… BINARY LOAD …]

CREATE TABLE flat (x,y,z,morton);

INSERT INTO flat SELECT * FROM flat_temp ORDER BY morton;

DROP TABLE flat_temp;

CREATE TABLE flat_temp (x,y,z,morton);[… LOAD …]

CREATE TABLE flat (x,y,z,morton), CONSTRAINT PRIMARY KEY (morton, z) ORGANIZATION

DROP TABLE flat_temp;


118/128


119/128

Research and others


120/128


Research and othersFaster flat tables: “Morton queries”

CREATE TABLE query_results_1 AS (SELECT x,y,z FROM (

SELECT x,y,z FROM ahn_flat WHERE ((morton2D between 1341720113446912 and 13417

OR (morton2D between 1341720126029824 and 13417OR (morton2D between 1341720310579200 and 13417OR (morton2D between 1341720474157056 and 13417OR (morton2D between 1341720482545664 and 13417OR (morton2D between 1341720671289344 and 13417

OR (morton2D between 1341720679677952 and 13417) a WHERE (x between 85670.0 and 85721.0)

and (y between 446416.0 and 446469.0));

Research and others


121/128


Research and othersFaster flat tables: “Morton queries”

0

1

2

3

4

5

6

20M 210M 2201M 23090M

T i m e [ s ]

Query #1 (rectangle 51 x 53 m, 74872 points)

PostgreSQL

PostgreSQL Morton

0

1

2

3

4

5

6

20M 210M 2201M 23090M

T i m e [ s ]

MonetDB

MonetDB Morton

Research and others


122/128


Research and othersFull AHN2 benchmark: loading

system Total loadtime [hours] Totalsize [TB] #points

LAStools unlic. 22:54 12.181 638,609,393,

LAStools lic 16:47 11.617 638,609,393,

LAStools lic LAZ 15:48 0.973 638,609,393,

Oracle Exadata 4:39 2.240 639,478,217,

MonetDB* 17:21 15.00 639,478,217,

Research and others


123/128


Research and othersFull AHN2 benchmark: querying

#points Time[s]

QUERY LAStools u nlic. LAStools Exadata MonetDB LAStools u nlic. L AS DB LAStools u nlic. L AS No DB LAStools L AS DB LAStools L AS No DB LAStools L AZ DB

01_S_RECT 74861 74863 74863 74855 0,07 0,79 0,06 0,77 0,202_L_RECT 718057 718070 718070 718132 0,16 0,85 0,16 0,78 0,49

03_S_CIRC 34700 34699 34675 34699 0,07 0,78 0,06 0,7 0,2704_L_CIRC 563119 563105 563082 563105 0,16 0,85 0,12 0,78 0,4410_S_RECT_MAXZ 2413 1734 2434 2434 0,08 0,82 0,08 0,69 0,4311_S_RECT_MINZ 591 591 591 591 0,05 0,84 0,05 0,72 0,1312_L_RECT_MINZ 343168 343168 343171 343171 0,26 1 0,26 0,98 1,3216_XL_RECT_EMPTY 0 0 0 0 0,04 0,75 0,04 0,66 0,0318_NN_1000 100019_NN_5000 500020_NN_1000_river 005_S_SIMP_POLY 182871 182875 182875 182875 0,7 31,05 0,39 28,2 0,6306_L_COMP_POLY_HOLE 460140 460158 387201 387193 1,52 32 1,57 30,2 1,9707_DIAG_RECT 45831 45813 45815 45815 0,55 31,43 0,55 28,76 0,9908_XL_POLYGON_2_HOLES 2365925 2365896 2273469 2273380 3,72 34,72 3,87 33,88 5,6709_S_POLYLINE_BUFFER 620568 620527 620719 620529 2,34 33,44 2,27 32,4 3,9415_XL_RECT 3992330 3992377 3992290 3992525 0,49 1,37 0,57 1,23 1,9417_XL_CIRC 2203066 2203055 2201280 2203055 0,32 1,16 0,32 1,01 1,121_L_NARROW_RECT 382395 382335 382335 386278 2,23 4,43 2,38 4,11 16,9427_STREET_DIAG_RECT 27443 27453 27459 27456 1,61 914,44 0,25 29,18 0,3513_L_POLYLINE_BUFFER 897042 896927 897359 897126 412,29 804,29 266,13 958,88 317,9114_DIAG_POLYLINE_BUFFER 765989 766039 766029 766029 102,19 419,78 104,09 428,51 151,4522_L_NARROW_DIAG_RECT 12148049 12147863 121478022246.9 154,97 1115,2 141,95 1800,41 284,5823_XL_NARROW_DIAG_RECT 691422551 6,91E+08 6,91E+08151.16 379,2 886,35 318,93 1051,76 552,0124_L_NARROW_DIAG_RECT_2 2468239 2468268 2468367 2468367 251,13 4106,54 294,99 13454 551,1325_PROV_DIAG_RECT 2,03E+09 3,53E+1412228.7626_MUNI_RECT 2124162497 2,12E+09 2,12E+09622.73 417,28 1127,56 214,98 405,26 536,128_VAALS 809097723 8,09E+08 8,67E+081469.47 1635,3 1883,19 2240,1 2525,98 2426,4329_MONTFERLAND 1509662615 1,51E+09 1,51E+095577.69 3579,73 5746,61 4809,9 6419,05 3855,9330_WESTERVELD 2,9E+09 1,34E+1427683.62

Research and others


124/128



0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

0,9

1

T i m e [ s ]

Easy queries

Research and others


125/128



0

0,5

1

1,5

2

2,5

3

3,5

4

4,5

5

T i m e [ s ]

Medium queries

Research and others


126/128



0

500

1000

1500

2000

2500

3000

3500

4000

T i m e [ s ]

Difficult queries

Program Agenda


127/128


Program Agenda

1

2

3

4

5

Introduction



Loading

Querying

Research and others

Conclusions

6

7

Conclusions


128/128

Conclusions

• For simple queries LAStools is the fastest but has not DBMS functionalities.

• Careful with multi-user environments

• LAStools with LAZ offers best storage (but slower queries)

• For complex queries DBs with blocks offer good performance

• For small datasets (million points) flat tables are suitable

– For Exadata, a lot more

• Blocks require packing/unpacking which adds overhead (noticeable in small queries)

• DB loading should be improved

• Parallel algorithms improve query response times

– Parallelism can be pushed further

• SFC-based flat tables improve scaling issue and faster queries (being investigated)

Documents

Workshop Dubai Cut