Scaling GIS Data in Non-relational Data Stores

Scaling GIS Data in Nonrelational Data Stores

featuring Mike Malone

Tuesday, March 30, 2010

Mike Malone@mjmalone



SimpleGeoScalable turnkey location infrastructure

Allows you to easily add geo-aware features to an existing application

That result: we need to store and query lots of data (data set is already approaching 1TB, and we haven’t launched)


Scaling HTTP is easyNo shared state - shared-nothing architecture• HTTP requests contain all of the information

necessary to generate a response

• HTTP responses contain all of the information necessary for clients to interpret them

• In other words, requests are self-contained and different requests can be routed to different servers

Uniform interface - allows middleware applications to proxy requests, creating a tiered architecture and making load balancing trivial


So what’s the problem? Individual HTTP requests have no shared state, but the applications that communicate via HTTP can and do

Application state has to live somewhere• Path of least resistance is usually a relational

database

• But RDBMSs aren’t always the best tool for the job


Desirable Data Store Characteristics

Massively distributed

Horizontally scalable

Fault tolerant

Fast

Always available


Relational DatabasesBased on the “relational model” first proposed by E.F. Codd in 1969

Tons of implementation experience and lots of robust open source and proprietary implementations


RDBMS StrenghtsTheoretically pure

Clean abstraction

Declarative syntax

Mostly standardized

Easy to reason about data


ACIDAtomicity - if one part of a transaction fails, the entire transaction fails

Consistency - all data constraints must be met for a transaction to be successful

Isolation - other operations can’t see a transaction that has not yet completed

Durability - once the client has been notified that a transaction succeeded, the transaction will not be lost


RDBMS WeaknessesSQL is opaque, and query parsers don’t always do the right thing• Geospatial SQL is particularly bad

The best ones are crazy expensive

Really bad at scaling writes

Strong consistency requirements make horizontal scaling difficult


RDBMS WritesRelational databases almost always use B-Tree (or some other tree-based) indexes

Writes are typically implemented by doing an in-place update on disk• Requires random seek to a specific location on

disk

• May require additional seeks to read indexes if they outgrow the disk cache

Disk seeks are bad.


CAP Theorem

There are three desirable characteristics of a shared data system that is deployed in a

distributed environment like the web.


CAP Theorem1. Consistency - every node in the system contains the same data (e.g., replicas are never out of date)

2. Availability - every request to a non-failing node in the system returns a response

3. Partition Tolerance - system properties (consistency and/or availability) hold even when the system is partitioned and data is lost


CAP Theorem

Choose two.


Node A Node B

Client

reads & writes

replicates

reads & writes


Node A Node B

Client

writes

replicates


Node A Node B

Client

responds

acknowledges


Node A Node B

Client

responds

o noes!


What now?

1. Write fails: data store is unavailable

2. Write succeeds on Node A: data is inconsistent


RDBMS ConsistencyRelational databases prioritize consistency

Large scale distributed systems need to be highly available• As we add servers, the possibility of a network

partition or node failure becomes an inevitability

We could write an abstraction layer on top of a relational data store that trades consistency for availability

Or we could switch to a data store that prioritizes the characteristics we really want


Nonrelational DBs

• CouchDB

• Cassandra

• Dynamo

• BigTable

• Riak

• Redis

• MongoDB

• SimpleDB

• Memcached

• MemcacheDB

Over the past couple years, a number of specialized data stores have emerged


Also Known As NoSQLNot entirely appropriate, since SQL can be implemented on non-relational DBs

But SQL is an opaque abstraction with lots of features that are difficult or impossible to efficiently distribute


So what’s different?Most “non-relational” stores specifically emphasize partition tolerance and availability

Typically provide a more relaxed guarantee of eventually consistent


NoACID


BASE

Basically Available

Soft State

Eventually Consistent


Eventual ConsistencyWrite operations are attempted on n nodes that are “authoritative” for the provided key

In the event of a network partition, data is written to another node in the cluster

When the network heals and nodes become available again, inconsistent data is updated


SimpleGeo ♥ CassandraNo single point of failure

Efficient online cluster rebalancing allows for incremental scalability

Emphasizes availability and partition tolerance• Eventually consistent

• Tradeoff between consistency and latency exposed to the client

Battle tested - large clusters at Facebook, Digg, and Twitter


Cassandra Data ModelColumn - a tuple containing a name, value, and timestamp

Column Family - a group of columns that are stored together on disk

Row - identifier for a specific group of columns in a column family

Super Column - a column that has columns


Cassandra Data Model{ '9xj5ss824mzyv.12345': { 'Record': { 'lat': 40.0149856, 'lon': -105.2705456, 'city': 'Boulder', 'state': 'CO' }, }, 'dr5regy3zcfgr.67890': { 'Record': { 'lat': 40.7142691, 'lon': -74.0059729, 'city': 'New York', 'state': 'NY' } }}








Writes are crazy fastWrites are written to a commit log in the order they’re received - serial I/O

New data is stored in an in-memory table

Memory table is periodically synced to a file

Files are occasionally merged

Reads may end up checking multiple files (bloom filter helps) and merging results• Thats okay because reads are pretty easy to scale


How can I query?Depends on the partitioner you use• Random partitioner: makes it really easy to

keep a cluster balanced, but can only do lookups by row key

• Order-preserving partitioner: stores data ordered by row key, so it can query for ranges of keys, but it’s a lot harder to keep balanced


BYOI• If you need an index on something other than

the row key, you need to build an inverted index yourself• Row key: attribute you're interested in plus row key

being indexed

• “dr5regy3zcfgr:com.simplegeo/1”

• But what about indexing multiple attributes..?


The Curse of DimensionalityLocation data is multidimensional

Traditional GIS software typically uses some variation of a Quadtree or R-Tree for indexes

Like B-Trees, R-Trees need to be updated in-place and are expensive to manipulate when they outgrow memory


Dimensionality ReductionIf we think of the world as two-dimensional cartesian plane, we can think of latitude and longitude as coordinates for that plane

Instead of using (x, y) coordinates, we can break the plane into a grid and number each box• Space-filling curve: a continuous line that

intersects every point in a two-dimensional plane



GeohashA convenient dimensionality reduction mechanism for (latitude, longitude) coordinates that uses a Z-Curve

Simply interleave the bits of a (latitude, longitude) pair and base32 encode the result

Interesting characteristics• Easy to calculate and to reverse

• Represent bounding boxes

• Truncating bits from the end of a geohash results in a larger geohash bounding the original


Geohash Drawbacks

Z-Curves are not necessarily the most efficient space-filling curve for range queries• Points on either end of the Z’s diagonal seem

close together when they’re not

• Points next to each other on the spherical earth may end up on opposite sides of our plane

These inefficiencies mean we sometimes have to run multiple queries, or expand bounding box queries to cover very large expanses


Geohash AlternativesHilbert curves: improve on Z-Curves but have different drawbacks

Non-algorithmic unique identifiers• Provide unique identifiers for geopolitical and

colloquial bounding polygons

• Yahoo! GeoPlanet’s WOEIDs are a good example


Other stuff we use


MemcacheUseful for storing ephemeral or short-lived data and for caching

Super crazy extra fast

Robust support from pretty much every language in the world


MemcacheDBBDB backed memcache

We use it for statistics• Can’t use Cassandra because it doesn’t

support eventually consistent increment and decrement operations (yet)

Giant con: it’s pretty much impossible to rebalance if you add a node


Pushpin ServiceCustom storage solution

R-Tree index for fast lookups

Mostly fixed data sets so it’s ok that we can’t update data efficiently


MySQL!

Our website still uses MySQL for some stuff... though we’re moving away from it


Thanks!


@mjmalone

Ask me questions!


Technology

Scaling GIS Data in Non-relational Data Stores