Spatial Data and Microsoft Azure SQL Database€¦ · WP, LightSwitch, WCF, ASP.Net MVC,, MS SQL...

Spatial Data and Microsoft

Azure SQL Database

Dealing with Spatial Data in Microsoft Azure

Azure SQL Database and Geospatial support

Mihail MateevSenior Technical Evangelist, Evangeism & Community Lead @ Infragistics

Welcome!

About me

• Mihail Mateev is a Senior Technical Evangelist, Team Lead at Infragistics Inc., Community Lead for Europe, PASS CEE Regional Mentor, Microsoft Azure MVP

• Mihail works in various areas related to Microsoft technologies : Silverlight, WPF, WP, LightSwitch, WCF, ASP.Net MVC,, MS SQL Server and Microsoft Azure

• More than 5 years GIS consultant and developer @ ESRI

Agenda

• What is Spatial Data

• Spatial Reference System

• Geospatial Features

• Spatial Data Types

• Creating Spatial Data

• Import Spatial Data

• Windows Azure SQL Database Architecture

• Migrating Spatial Data to SQL Azure

• Using Spatial Data in Windows Azure SQL Database

What is Spatial Data

• Spatial data describes the position, shape and orientation of objects in space

What is Spatial Data

• Analyzing sales trends

• Analyzing the best placement depending of different criteria

• Navigating to s destination using a GPS device

• Allowing customers to track the deliveries

• Finding the optimum route for transportation tasks

• Reporting geospatial information on the map rather than in a tabular or chart format

Spatial Reference Systems

• Define positions on a three-dimensional, round model of the earth

Spatial Reference Systems

• Describe the position of points on the earth surface as the lie on a flat, two-dimensional plane.

- X+ Y

- X- Y

+ X- Y

X

+ X+ Y

Data

usually here

Y

Spatial Reference Systems

• Datum is a set of reference points on the Earth's surface against which position measurements are made

Earth surface

Local datum NAD27Ellipsoid CLARKE 1866

Earth-centered datum NAD83 Ellipsoid GRS80

Center of the Earth's mass

+

* *

Spatial Reference Systems

• Reference ellipsoid– An approximation of the surface shape of the earth (or

rather, the geoid) spheroid used for the needs of geodesy at some of the earth's surface

Geospatial Features

• Points

• Polylines

• Polygons

Points Polylines Polygons

Geospatial Features

• Vector Data:

– describes discrete spatial objects by defining the coordinates of geometries that approximate the shape of those features

X,Y

X,Y X,Y X,Y

Geospatial Features

• Raster Data:

– represents spatial information using a matrix of cells. These cells are arranged into a grid that is overlaid onto the surface of the earth.

X,Y

Rows

Columns

SQL Server Spatial Data Types

SQL Server supports two different spatial data types

• GEOMETRY and GEOGRAPHY

Shapefile format

Donut.dbf table

Shape field accesses separate coordinate files

Donutshapefile

Spatial Data Types

• Spatial data and SQL Server

Point

MultiPoint

LineString

MultiLineString

Polygon

MultiPolygon

GeometryCollection

Creating Spatial Data

• Creation of spatial objects

Creating Spatial Data

• Creation of spatial objectsCreating instances of UDT (User Defined Types)

• Geometry and Geography data types are implemented as user-defined types (User Defined Types, UDT), written on. NET. They are automatically installed with the server and are available for use in any database, SQL Server.

• Any variable, parameter, or column of a table can be declared as the type of Geometry. Note that the type name is not case sensitive.

• DECLARE @g Geometry

Creating Spatial Data

• The use of spatial data

Use STGeomFromText () to create a LineString

• DECLARE @g Geometry

• SET @g = Geometry::STGeomFromText('LINESTRING(0 0, 10 10, 21 2)', 0)

Creating Spatial Data

• The use of spatial dataUse STGeomFromText () and STPointFromtext ()

• DECLARE @Glasgow as geography

• SET @Glasgow = geography ::STPointFromText('POINT(258647 665289)', 27700)

• DECLARE @Glasgow2 as geometry

• SET @Glasgow2 = geometry::STGeomFromText('POINT(258647 665289)', 27700)

Creating Spatial Data

• The use of spatial data

Z and M coordinates

• Z coordinate represents the height or the height of a point.

• M coordinates are stored "measure" value of point. These coordinates can be used to provide additional details of a point, which can be expressed as a double-precision

• Imagine WKT from the Z and M coordinates:

• POINT (x y z m) or PONT (longitude latitude z m)

Creating Spatial Data

• The use of spatial data

Management of spatial data

CREATE TABLE SpatialTable

(Id int IDENTITY (1,1),

GeomCol1 geometry,

GeomCol2 AS GeomCol1.STAsText ());

GO

INSERT INTO SpatialTable (GeomCol1)

VALUES (geometry :: STGeomFromText ('LINESTRING (100 100, 20 180, 180 180), 0));

Creating Spatial Data

• The use of spatial data

Management of spatial data

• STBuffer

SELECT @g.STBuffer(8), @g.STBuffer(8).ToString()

• STExteriorRing

SELECT @g.STBuffer(8).STExteriorRing(), @g.STBuffer(8).STExteriorRing().ToString()

Creating Spatial Data

• The use of spatial data

Management of spatial data• STArea, STLength

DECLARE @g GEOMETRY = 'POLYGON((10 10, 10 40, 40 40, 10 10))'

SELECT @g.STArea() as Area, @g.STLength() as Length

Creating Spatial Data

• Using Stored Procedures

CREATE PROCEDURE [dbo].[get_SelectedData] @pCntrNameValuenvarchar(255)

BEGIN

SET NOCOUNT ON;

SELECT geom.STBuffer(2) as geom, CNTRY_NAME, POP_CNTRY FROM world WHERE CNTRY_NAME = @pCntrNameValue

END

Indexing

• The Need for a Spatial Index

• How Does a Spatial Index Work?

• The Primary and Secondary Filter

– Primary filter:

– Secondary filter:

Indexing

• The Grid Structure of a Spatial Index

Indexing

• The Grid Structure of a Spatial Index

Indexing

• Optimization Rules

– a Multilevel Grid Index

• Covering Rule

Indexing

• Using Spatial Indexes:

USING GEOMETRY_GRID WITH (

BOUNDING_BOX = (0, 0, 4096, 4096),

GRIDS = (

LEVEL_1 = MEDIUM,

LEVEL_2 = MEDIUM,

LEVEL_3 = MEDIUM,

LEVEL_4 = MEDIUM),

CELLS_PER_OBJECT = 16);

Indexing

• Queries to Use a Spatial Index

– Supported Methods• Filter()

• STContains()

• STDistance()

• STEquals()

• STIntersects()

• STOverlaps()

• STTouches()

• STWithin()

Indexing

• Using Spatial Indexes:

CREATE TABLE Points (

id char(1) NOT NULL,

shape geometry

);

DECLARE @Polygon geometry = 'POLYGON ((1.5 0.5, ...))';

SELECT id

FROM Points WITH(INDEX(sidxPoints))

WHERE shape.STIntersects(@Polygon) = 1;

Indexing

• Using Spatial Indexes:

CREATE TABLE IndexTest (

id int NOT NULL,

geom geometry,

CONSTRAINT pk_IndexTest PRIMARY KEY CLUSTERED (id ASC)

);

CREATE SPATIAL INDEX sidx_IndexTest ON IndexTest(geom)

WITH ( BOUNDING_BOX = (0, 0, 10, 10) );

• SELECT * FROM IndexTest

• WHERE geom.STIntersects('POINT(3 2)') = 1;

Indexing

• Using Spatial Indexes:

ALTER TABLE IndexTest ADD geom_length AS geom.STLength() PERSISTED;

CREATE INDEX idx_geom_length ON IndexTest(geom_length);

SELECT * FROM IndexTest

WHERE geom.STLength() > 100;

Indexing

• Adding an Index Hint:

SELECT * FROM IndexTest WITH(INDEX(sidx_IndexTest))

WHERE geom.STIntersects('POINT(3 2)') = 1;

Creating Spatial Data

• Using Stored Procedures

Import Spatial Data

• SQL Server import and export spatial data formats

Well Known Text,

Well Known Binary

Geographic Markup Language (GML)

Shapefile

Import Spatial Data

• Import spatial data in SQL Server

Conversion Utilities:

• ShapeToSQL: www.social.msdn.microsoft.com

• Shp2text: www.obviously.com

• FME : www.safe.com

• Manifold: www.manifold.net

Microsoft Azure SQL Database

Architecture

• Architecture• There are four different levels of

abstraction, which work together to provide a relational database for your application:

Client layer

Services layer

Platform layer

Infrastructure layer

• SQL Azure offers only DTS endpoint

• For each Windows Azure SQL database instance there was created 3 copies of SQL

Windows Azure SQL Database

Architecture

• Azure SQL Database Disadvantages

Azure SQL Database does not support the following functions of the database

• Service Broker

• HTTP access

• CLR stored procedures

Microsoft Azure SQL Database

Architecture

• Azure SQL Database Disadvantages

Azure SQL Database support spatial data types, but developers need to deploy libraries with spatial CLR types with their applications. Windows Azure has no installed predefined SQL spatial data types

Azure SQL Database does not support the following functional features• Distributed queries

• Distributed transactions

• Any SQL query and views that change or get physical information resources

Microsoft Azure SQL Database

Architecture

• Azure SQL Database Spatial Data Support

Microsoft Azure SQL Database supports enhancements to spatial data types:

• new and updated methods and aggregates for geometry and geography;

• improved precision, enhancements to the geography type;

• spatial performance improvements;

• spatial helper stored procedures;

• support for persisted computed columns;

• changes in the client-side spatial programming library.

Microsoft Azure SQL Database

Architecture

• Azure SQL Database Spatial Data Support

Microsoft Azure SQL Database spatial data types issues:prior to 2012 Azure SQL Database didn’t support :

• Most of stored procedures, related to spatial types

• Most of Boolean operations

• Spatial aggregates didn’t work correectly

Azure SQL Database Federations

• DB Scaling solutions: Scale UP vs. Scale OUT

Scale UP:

• This approach relates to improving the hardware solution of the DB - faster processor, more RAM, faster Hard Drive

• A single SQL Azure database can contain up to 150GB

• NewDB Editions in Azure (Preview):

– Standard (250 GB)

– Premium (500 GB)

Azure SQL Database Federations

• DB Scaling solutions: Scale UP vs. Scale OUT

Scale OUT:

• The Scale OUT approach relates to implementing a DB solution in which the I/O process will be distributed across multiple DB partitions.

• Scale OUT options:

– Table partitioning

–Master/Slave configurations

– Cluster Computing

– Sharding

Azure SQL Database Federations

• DB Scaling solutions: Scale UP vs. Scale OUT Scale OUT:

• Table partitioning

– a large table is split into two or more physically separate partitions

• Master/Slave configurations

– one database which is called “Master” and multiple databases called “Slaves”.

• Cluster Computing

– Similar to the Master/Slave. In such scenario one of the read-only nodes becomes the new “Master” if the “Master” fails

• Sharding

Azure SQL Database Federations

• Scale UP vs. Scale OUT Scale OUT:

• Sharding:

– “Sharding” or “Shared nothing”

– the application operates with “Shards” which can be physically separate databases.

– there are queries which run inside the shard and queries which are distributed across multiple shards (you should minimize it)

Azure SQL Database Federations

• Sharding SQL Azure Federations:

• Azure Federation is a Shardingtechnology which allows distributing the DB transactions across multiple databases which are called Federation Members.

– Federation object

– Federation Key

– Federation Member

– Atomic Unit

Azure SQL Database Federations

• Sharding SQL Azure Federations:

– Federation Root: Refers to the database that houses federation object.

– Federation Distribution Key: This is the key used for data distribution in the federations.

– Federation Atomic Unit: Represent all data that belongs to a single instance of a federation key.

– Federated Tables: Refer to tables that contain data that is distributed by the federation. Federated tables are created in federation members and contain a federation distribution key annotated with the FEDERATED ON

Azure SQL Database Federations

• Horizontal Partitioning and Vertical Partitioning

– Horizontal partitioning, likes you use a knife to cut the table horizontally, which means split the table by rows

– Vertical partitioning means split the table by columns

Azure SQL Database Federations

• SQL Azure Federations

– utilize the horizontal partitioning to split the tables in multiple databases.

– make sure that all records in the tables that referred to the same ID must be in the same partition

Azure SQL Database Federations

• SQL Azure Federations

A federated database contains three types of table:

– Federated

– Reference

– Common

A federated table is created by appending FEDERATED ON to a CREATE TABLE statement.

CREATE TABLE (…)

FEDERATED ON (Customerid = custId)

Azure SQL Database Federations

• SQL Azure Federations and Spatial Support

– Geography and geometry types cannot be used as the data type of the column that a table is federated on; however they can be part of the federated table. There are no other limitations on using spatial data with federations.

– After a SPLIT or DROP operation, spatial indexes stay consistent and intact in the destination federation members

Migrating Spatial Data to SQL

Azure• SQL Server 2008/2008 R2

Microsoft SQL Management Studio for SQL Server 2008/2008 R2 does not support the transfer of spatial data in Azure SQL Database

• SQL Azure Migration Wizard: http://sqlazuremw.codeplex.com/

Migrating Spatial Data to SQL

Azure

• SQL Server 2012 / 2014

Microsoft SQL Management Studio for SQL Server 2012 / 2014 does supports the transfer of spatial data in Azure SQL Database

Migration of systems with spatial

data in Azure SQL Database

Create a new Azure SQL database

Database migration – migrate your databse from the local SQL Server to SQL Azure

Add a new Windows Azure Storage Account and Web services

Add web project role in the decision

Change settings in the Web.config and ServiceReferences.ClientConfig

Publish Windows Azure Cloud Service

Using Spatial Data in Microsoft

Azure SQL Database

• Processing of spatial data using .NET in Windows Azure

SqlCommand sqlCommand = new SqlCommand();

sqlCommand.Connection = sqlConnection;sqlCommand.CommandType = System.Data.CommandType.StoredProcedure;sqlCommand.CommandText = "get_WorldData";sqlConnection.Open();

SqlDataReader reader = sqlCommand.ExecuteReader();

if (reader != null)while (reader.Read()){

Dictionary<string, string> valueDictionary = new Dictionary<string, string>();for (int i = 0; i < reader.FieldCount; i++){

valueDictionary.Add(reader.GetName(i), reader.GetValue(i).ToString());}list.Add(valueDictionary);

}....

Using Spatial Data in Microsoft

Azure SQL Database

• Using Entity Framework 5 and later versionsList<SpatialPoint> multiPoints = new List<SpatialPoint>();

var numPoints = country.geom.Envelope.ElementAt(1).PointCount;

for (int i = 1; i <= numPoints; i++)

{

SpatialPoint pnt = new SpatialPoint((double)(country.geom.Envelope.ElementAt(1).PointAt(i).XCoordinate), (double)(country.geom.Envelope.ElementAt(1).PointAt(i).YCoordinate));

multiPoints.Add(pnt);

}

SpatialRect rect = multiPoints.GetBounds();

Using Spatial Data in Microsoft

Azure SQL Database

• Windows Azure Web Site and spatial data

Creating a Windows Azure Web Site

Using Spatial Data in Microsoft

Azure SQL Database

• What libraries to use?

Microsoft.SqlServer.Types.dll• (managed) .NET library

• \Program Files(x86)\Microsoft SQL Server\110\SDK\Assemblies

SQLServerSpatial.dll / SQLServerSpatial110.dll• (unmanaged) C++ library

• \Windows\System32

Using Spatial Data in Microsoft

Azure SQL Database

• Getting libraries

Project –> Add Existing Item

\Windows\SysWOW64 -> msvcp110.dll and msvcr110.dll files (Visual Studio 2012)

or msvcp100.dll и msvcr100.dll - required for spatial data(Visual Studio 2010) .

Using Spatial Data in Microsoft

Azure SQL Database

• Getting libraries Set the properties of msvcp100.dll, msvcr100.dll and

SqlServerSpatial.dll / SqlServerSpatial110.dll to “Copy to Output directory = Copy always”

Using Spatial Data in Microsoft

Azure SQL Database

• Microsoft Azure and Entity Framework

Deploying Spatial Data Types to Microsoft Azure

• In order to use spatial data types on Windows Azure you will need to deploy the Microsoft.SqlServer.Typesassembly with your application. The easiest way to do this is to install the Microsoft.SqlServer.Types NuGetpackage in your application.

PM> Install-Package Microsoft.SqlServer.Types

Using Spatial Data in Microsoft

Azure SQL Database

• Microsoft Azure and Entity Framework

SQL Azure Federations

• The current release of Entity Framework can be used to work with SQL Azure Federations, however a federated database cannot be created by the Entity Framework.

Demo

Spatial Data and Microsoft Azure

Storage

• Is Azure Storage NoSQL ?

• Using spatial data in the Microsoft Azure Blob Storage

• Accessing the Windows Azure Storage is done via a storage account. A storage account can have many blob containers.A container is a user-defined set of blobs that has only properties, which include a list of the blobs it contains. Containers don’t store data directly.

Spatial Data and Microsoft Azure

Storage

• Windows Azure Blob Storage

• Windows Azure Blob Storage Concepts

Spatial Data and Microsoft Azure

Storage

• Microsoft Azure Blob Storage

Blob URL • http://<Account>.blob.core.windows.net/<Container>/<BlobName>

URL: • http://sally.blob.core.windows.net/music/rock/rush/xanadu.mp3

Types of Blobs

– Block Blob

– Page Blob

Spatial Data and Microsoft Azure

Storage

• Steps to Implement Azure Application Reading Shapefiles from Blob Storage

– Implement a WCF to upload and download files, using Windows Azure Blob Storage.

– Add in the client application (ASP.Net / Silverlight ) XamMapcomponent.

– Implement shapefile loading from Isolated Storage to XamMap.

– Create a new Windows Azure Storage Account and a Hosted Service (if you are using a Windows Azure Account).

– Add a Web Role project in Solution

– Edit settings in ServiceReferences.ClientConfig .

– Publish the Windows Azure Cloud Service

Useful links

– Shape To SQL Tools http://www.sharpgis.net/page/SQL-Server-2008-Spatial-Tools.aspx

– SAFE Softwarehttp://www.safe.com/

– Infragistics Blogshttp://blogs.infragistics.com/

www.infragistics.com/mihail_mateev/

Welcome to SQLSaturday #311

SQLSaturday is a training event for Microsoft Data Platform11 of October, 2014

110 B, Simeonovsko Shosse Bul, Sofia

5 tracks:Development

DBAMicrosoft Azure

Open Source SoftwareBI

http://www.sqlsaturday.com/311/

Thank you!

Q & A