ExtremeScaling with SQL

AzureSQL Bits 7, York, October 2010Martin Schmidt – Miracle A/S

Denmark

Subtitle:The Safetrack story

AgendaO Speaker InfoO Safetrack facts & system infoO Safetrack architectureO Database designO Issues with current setupO The Azure platformO ShardingO Solution WalkthroughO Moving Exsisting Data to SQL Azure

Speaker infoO Name: Martin SchmidtO Email: msc@miracleas.dkO Blog: www.performanceduo.comO Twitter: @ms1333

AgendaO Speaker InfoO Safetrack facts & system infoO Safetrack architectureO Database designO Issues with current setupO The Azure platformO ShardingO Solution WalkthroughO Moving Exsisting Data to SQL Azure

Safetrack factsO Danish Company – started 2005O GPS tracking of trucks and excavatorsO Started out as an insurance service

against theftO Today: A complete platform for fleet

managementO Tracks more than 25.000 unitsO Website: http://www.safetrack.dk

Unit Types

How does this Work?

How does this WorkO All units are equipped with a GPS

transmitterO Sends binary packages over GPRSO Built-in SIM cardO Sends a package every 2 min when

runningO Sends a package every 15 min when

parkedO Data is sent using UDP Protocol

ME - 301

Length = 75 mm Width = 45 mmHeight = 18 mm

Package dataO ReporttimeO LongitudeO LatitudeO SpeedO KM StatusO Inputs status information

O Is the lift up / downO Is the engine running

O Timecounter info on inputsO For how long has the engine been runningO How long has the lift been in use

AgendaO Speaker InfoO Safetrack facts & system infoO Safetrack architectureO Database designO Issues with current setupO The Azure platformO ShardingO Solution WalkthroughO Moving Exsisting Data to SQL Azure

System Archietecture

AgendaO Speaker InfoO Safetrack facts & system infoO Safetrack architectureO Database designO Issues with current setupO The Azure platformO ShardingO Solution WalkthroughO Moving Exsisting Data to SQL Azure

Database Design

AgendaO Speaker InfoO Safetrack facts & system infoO Safetrack architectureO Database designO Issues with current setupO The Azure platformO ShardingO Solution WalkthroughO Moving Exsisting Data to SQL Azure

Issues ?O Database and application on same

boxO Everything runs on the same single

serverO Single Point of Failure

O HardwareO DatabaseO Services / Website

O No HA solutionO Limited scaling options (Buy bigger

HW)

Issues ?O All elements are independendO Easy to split thingsO No ”secret” data

O No personal or banking informationO Not breaking the Danish data law

Issues ?O SQL Azure in the future?O Windows Azure in the future?

O Why Not

AgendaO Speaker InfoO Safetrack facts & system infoO Safetrack architectureO Database designO Issues with current setupO The Azure platformO ShardingO Solution WalkthroughO Moving Exsisting Data to SQL Azure

SQL AzureO High Availibity = Free

O 1 DB = 3 DB’s (automatic failover)O Size limitations

O 1 – 50 GBO Safetrack database > 250 GB

O Get the database off the current boxO Release more power to the Windows

service

SQL AzureO Let Microsoft be your primary DBA!O SQL Azure = Black boksO Running your database with other

unknown applications

Windows AzureO Moving the website to Windows

AzureO Easy peasy

O The Windows Service is not compatible with Azure at presentO No support for UDP – TCP Only

SQL AzureO Lets move the database to the Cloud

O Latency from the local Windows service to SQL Azure is not a big issue.

O Size is a issue, we will have to Shard the data across multiple databases

O Sharding gives us scalability

AgendaO Speaker InfoO Safetrack facts & system infoO Safetrack architectureO Database designO Issues with current setupO The Azure platformO ShardingO Solution WalkthroughO Moving Exsisting Data to SQL Azure

Sharding factsO Partition data across multiple databasesO Each database holds a subset of the dataO A query needs only to look in one

databaseO Sharding is no silver bullet, you still need

to thinkO Important to choose the correct sharding

keyO Try to avoid fan-out queries, if possible

AgendaO Speaker InfoO Safetrack facts & system infoO Safetrack architectureO Database designO Issues with current setupO The Azure platformO ShardingO Solution WalkthroughO Moving Exsisting Data to SQL Azure

Solution walkthroughO Shard key candidates

O UnitIDO CustomerID

O The winner is:O CustomerID

O All reports on the web front end is based on customerID. This gives the customer the option to se the status of all his units.

Data SyncronisationO MS Data sync Framework

O Requires SQL Agent to do scheduling

O SQL Azure Data Sync (Dev Preview)O The winner so farO We will have a look in a minuteO Pretty Slow

O SQL Server Integration ServicesO Still no scheduler

O Windows Azure worker roleO We need to test this

Database Design

DEMOO Enough Power Pointing

AgendaO Speaker InfoO Safetrack facts & system infoO Safetrack architectureO Database designO Issues with current setupO The Azure platformO ShardingO Solution WalkthroughO Moving Exsisting Data to SQL Azure

Loading data into Azure

O Task: Load existing data into SQL AzureO 265.000.000 Rows (ReportMessage)O 150 GB

O Possible solutionsO BCP UtilityO SSIS

Loading data into Azure

Loading data into Azure

O Slow, poor performanceO Still working on improving

performance

O Tested throughput (Laptop)O 1000 Rows / sec

SQL Azure RessourcesO SQL Azure webfrontendO http://sql.azure.com

O SQL Azure LabsO http://www.sqlazurelabs.com

O SQL Azure team BlogO http://blogs.msdn.com/b/sqlazure/

More RessourcesO PerformanceDUO

O http://www.performanceduo.com

O Miracle A/SO http://www.miracleas.dk

O MiracleDBO http://Miracledb.codeplex.com

O SafetrackO http://www.safetrack.dk

Q & AO Questions?

Thank You

Coming up…P/X001Understanding and Preventing SQL Injection AttacksKevin KlineP/L001SSIS FieldnotesDarren GreenP/L002The (Geospatial) Shapes of Things to ComeSimon MunroP/L005End to End Master Data Management with SQL Server Master Data ServicesJeremy KashelP/T007Understanding Microsoft Certification in SQL ServerChris Testa-O'Neill

#SQLBITS