Embed Size (px)
DESCRIPTION
presentation at the Big DataShow as part of InternetWorld in London April 2013 its an overview of the main technologies used to process big data, and then a review of some real world big data use cases - showing how they map onto the axes of complexity of analytics and responsiveness
Citation preview
When Two Seconds Is Too Long
A look at low latency, real-time analytics
VP Product [email protected]
@daiclegg
dai clegg
@daiclegg
2
The new data sources driving big data analytics
2
mobile marketingsocial appsinfrastructure monitoring
batch reportingexploratory analysisdata discovery
Batch AnalyticsOperational Intelligence
infrastructure fabric/logssmart grids/smart metersdfid tags, etc
social mediamobile appsweb clicks
Machine DataSocial Data
@daiclegg
TITLE HERE
3
the Big Data technology landscape
data mining/ data warehousing
quantitative analytics
data access
@daiclegg
Subtitle
TITLE HERE
4
the Big Data technology landscape
hours minutes seconds milli-secondsimmediacy of results
complexity of analytics
use case
money-offcoupons
data mining/ data warehousing
quantitative analytics
data access
‣ Identifies items that shoppers are likely to buy in future visits
‣ Coupon redemption rates as high as 24%
@daiclegg
Subtitle
TITLE HERE
babies
5
the Big Data technology landscape
hours minutes seconds milli-secondsimmediacy of results
money-offcoupons
data mining/ data warehousing
quantitative analytics
data access
complexity of analytics
use case
‣ Neo-natal infant monitoring‣ 120 babies monitored‣ 120k messages /second
@daiclegg
Subtitle
TITLE HERE
babies
6
the Big Data technology landscape
hours minutes seconds milli-secondsimmediacy of results
money-offcouponswindmills
data mining/ data warehousing
quantitative analytics
data access
complexity of analytics
use case
‣ 2.5 petabytes in Hadoop‣ weather data, turbine operational data‣ model weather to optimise wind farms
@daiclegg
Subtitle
TITLE HERE
babies
7
the Big Data technology landscape
hours minutes seconds milli-secondsimmediacy of results
money-offcouponswindmills
data mining/ data warehousing
quantitative analytics
data access
‣ Neo-natal infant monitoring‣ 120 babies monitored‣ 120k messages /second
‣ Global e-store‣ Shopping cart session store‣ 000s of transactions per second
shopping carts
complexity of analytics
use case
@daiclegg
Subtitle
TITLE HERE
babies shopping
carts
8
the Big Data technology landscape
hours minutes seconds milli-secondsimmediacy of results
money-offcouponswindmills
data mining/ data warehousing
quantitative analytics
data access
Hi-tech Mftg
‣ automated hi-tech assembly
‣ ‘000s of test readings per second‣ comparing results to historic metrics‣ monitoring the test stations’ performance
complexity of analytics
use case
@daiclegg
Subtitle
TITLE HERE
babies shopping
carts
9
the Big Data technology landscape
hours minutes seconds milli-secondsimmediacy of results
money-offcouponswindmills
data mining/ data warehousing
quantitative analytics
data access
Hi-tech Mftg
taxis
complexity of analytics
use case
‣ Neo-natal infant monitoring‣ 120 babies monitored‣ 120k messages /second
‣ Real-time visibility of infrastructure‣ Insight delivered into the cab‣ Caught competitor ‘stealing’ web data
@daiclegg
Subtitle
TITLE HERE
babies shopping
carts
10
the Big Data technology landscape
hours minutes seconds milli-secondsimmediacy of results
money-offcouponswindmills
data mining/ data warehousing
quantitative analytics
data access
Hi-tech Mftg
taxisSMS Mktg
‣ ingesting 300m SMSs daily‣ over 5000 events per second
‣ maintaining 90 days history per campaign‣ Oracle/NetApp only supported 45 days
complexity of analytics
use case
@daiclegg
Subtitle
TITLE HERE
babies shopping
carts
11
the Big Data technology landscape
hours minutes seconds milli-secondsimmediacy of results
money-offcouponswindmills
data mining/ data warehousing
quantitative analytics
data access
Hi-tech Mftg
taxisSMS Mktg
X-factor
complexity of analytics
use case
‣ Neo-natal infant monitoring‣ 120 babies monitored‣ 120k messages /second
‣ 1000s of votes/boos/applauds per second‣ “we were able to handle our problems with
$5,000 and a credit card.” - CTO, Tellybug
@daiclegg
Subtitle
TITLE HERE
babies shopping
carts
12
the Big Data technology landscape
hours minutes seconds milli-secondsimmediacy of results
money-offcouponswindmills
latest data, historic context, analytic insight
deep analyticsworth the wait
data mining/ data warehousing
quantitative analytics
data access
Hi-tech Mftg
taxisSMS Mktg
X-factor
just need put & get
just need recent data
complexity of analytics
use case
@daiclegg
the Acunu Analytics
13
Acunu
Analytics
delay text
delay text
Prod 4Prod 3Prod 2
Ventas Ron Rate
Ventas BSF
Ventas x VendedorMes en Curso
Cuentas Por Cobrar
Cuentas Por Pagar
Contabilidad
Prod 1
Acunu Analytics can ingest data, at very high velocity, from any source
The data is pre-processed, as it arrives, to filter, transform and enrich it with other corporate data
And aggregated into roll-up cubes of sums, averages, top k, etc, so query answers are already stored
Then, when a dashboard query is executed, the answer is there for instant response
But not just dashboards; there’s a JSON API so queries can be embedded in other apps
And the original data is stored for further analysis and to share with other analytic tools
