Mapping time - NYHOLTgijs.github.io/images/cartodb_datacubes.pdfworld, one project at a time. From data visualization to citizen science, it is all about stories that matter RECENT

Mapping timeusing CartoDB to enable bigdata(™) on the client

Andrew Hill & Simon Tokumine

Friday, 28 September 12

GeneralistVizzuality


Work Products Clients About Blog

Working to improve ourWorking to improve ourworld, one project at a time.world, one project at a time.From data visualization to citizen science, it is all about stories that matterFrom data visualization to citizen science, it is all about stories that matter

RECENT WORK

View more projects

LATESTLATEST Subscribe to RSS

Our project Evolution of the Web was declared Project of the Day by the FWA. Check it!

Evolution Of Web 2.0Google

Endangered LanguagesGoogle

GFW 2.0 PrototypeWorld Resources Institute

Irekia 2.0Basque Government

Carbon CalculatorUNEP-WCMC

OldWeatherZooniverse

PlanetHuntersZooniverse

ProtectedPlanetUNEP-WCMC

NGO Aid MapInteraction

Red List Geospatial ToolKew Gardens

NEEMO MISSIONNASA

Antarctic Field GuideSCARMARBIN

Wed, 12 Sep 2012The Graphical Web:Advanced mappingtechniques for real projectsWe’ve been today in Zurich at TheGraphical Web, an amazingconference for discussing “what isaesthetically possible on the web”.@saleiva has participated in asession on 2D Web Mapping andWeb-GIS ...Read more

Mon, 30 Jul 2012The Old Weather project iscomplete - 1.6 Million newobservationsThe Old Weather Project, a citizenscience initiative by Zooniverse, hasreached its goal: all the logbooks from280 World War I Royal Navy shipshave been read and transcribed intheir entirety. This...Read more

Mon, 09 Jul 2012Evolution of the Web 2.0Two weeks ago we launched TheEvolution of the Web, a revamped,dynamic visualization of the history ofweb technologies and browsers, andthe growth of global internet usersand traffic since the ear...Read more

Map data ©2012 Google, Tele Atlas - Terms of Use

We are an open source company

[email protected]

Come and meet us at Madrid office, we are only2 hours away by plane

148 Lafayette Street,Penthouse, NY 10013

+1 347 460-5284

NEW YORK

c/Hortaleza, 48, 1.28004, Madrid - Spain

+34 659 157 916

MADRID

Works Products Clients About Blog


member of Vizzuality. Small agency. Madrid and NYC. Data visualization. Working on stories that matter. Here to talk about CartoDB and some of the cool things you can do with it.

Same old thingsServer configGeospatial Database (backups, optimal config)Import/exportAdmin UI and configTiler + interactivity APIData APICachingAuthentication...


hosted geospatial database (postgis) + nice gui to take the grunt work out of our geospatial projects


hosted geospatial database (postgis) + nice gui to take the grunt work out of our geospatial projects




CartoDB APIsMaps API SQL API


SQL APIHTTP data API Full postgis 2.0R/W via API keysPublic accessRead only cache layerCDN'able


SQL API“select * from quake”

https://viz2.cartodb.com/api/v1/sql/?q=select%20*%20from%20quake&format=geojson








SQL API


_I want to see ‘X’ over timeusing the SQL API with big data


_Traditional approachDownload the data to the client and render it.http://goo.gl/c6KxN - Recent earthquakeshttp://goo.gl/B7NK6 - Syria casualties


http://goo.gl/c6KxN

http://goo.gl/c6KxN

http://goo.gl/B7NK6

http://goo.gl/B7NK6


_n-GB? n-Million rows?the problem



“The main limiting factor we've noticed with webgl is the bottleneck in bandwidth to get all the data over.”Josh Livni & Chris Broadfoot - Google. SXSW 2012.

http://www.youtube.com/watch?v=zH5bJSG0DZk&feature=youtu.be&t=2h44m20s








“Even though you can bring big data into the client you might not want to take all of it. There's definitely a case for taking slices of it”


Josh Livni & Chris Broadfoot - Google. SXSW 2012.








_n-GB? n-Million rows?redefining the problem


_Acceptable granularity?redefining the problem


Maybe Josh and Chris were talking about windowing/streaming, but maybe they were talking about datacubes.

Datacubes


Datacubes•BI/Finance analysis•multidimensional•summary data•binned variables


Aka. Hypercubes/OLAP cubes (Make city people sound like they're lawnmowerman)

DatacubesCons•can be lossy•take time to generate

Pros •fast access•lightweight•cachable


Aka. Hypercubes/OLAP cubes (Make city people sound like they're lawnmowerman)

_Can we cube the data?rather than tile it?


Real world usage. Can we replace tiles with these?


WITH hgrid AS (SELECT Cdb_rectanglegrid(Cdb_xyz_extent(8, 12, 5), Cdb_xyz_resolution(5) * 4, Cdb_xyz_resolution(5) * 4) AS cell) SELECT x, y, Array_agg(c) vals, Array_agg(d) dates FROM (SELECT St_xmax(hgrid.cell) x, St_ymax(hgrid.cell) y, Count(i.cartodb_id) c, Floor(( Date_part('epoch', built) -‐ -‐10418716800 ) / 32837875) d FROM hgrid, us_po_offices i WHERE St_intersects(i.the_geom_webmercator, hgrid.cell) GROUP BY hgrid.cell, Floor(( Date_part('epoch', built) -‐ -‐10418716800 ) / 32837875) ) f GROUP BY x, y


{ rows: [ { x: 0, y: 0, vals: [2], dates: [457] }, { x: 1, y: 0, vals: [1,1,4], dates: [2,3,4] } ]}


1

10

100

1000

3mb 70mb 300mb1.5

21.2

300

70

3

Raw Datacube

Payload sizesFriday, 28 September 12

_Thanks!Checkout www.cartodb.com for more!


http://www.cartodb.com

http://www.cartodb.com

Documents

Mapping time - NYHOLTgijs.github.io/images/cartodb_datacubes.pdfworld, one project at a time. From data visualization to citizen science, it is all about stories that matter RECENT