Michael newberry

Extracting Value from Big Data in the Cloud -

Michael Newberry

Big data in a Hybrid-Cloud worldDr Michael Newberry

Windows Azure Lead, Microsoft UKMichael.Newberry@Microsoft.com

Doggerland: Simon Fitch, Vince Gaffney and Ken ThomsonImage Source: drowned-landscapes.tumblr.comRoyal Society's Summer Science Blog (http://summer-science.tumblr.com/)

Big Data.

Big Data.

VOLUME (Size)

VARIETY (Structure)

VELOCITY (Speed)

Getting useful insightsfrom awkward data setsusing the most appropriate computing platform at each stage.

Dr Michael NewberryWindows Azure LeadMicrosoft UK

Big data in a Hybrid-Cloud worldDr Michael Newberry

Windows Azure Lead, Microsoft UKMichael.Newberry@Microsoft.com

Machine Learning & Bayes theorem

𝑝 ( h𝑀𝑖𝑐 𝑎𝑒𝑙𝑏𝑢𝑦𝑖𝑛𝑔𝑠𝑜𝑐𝑘𝑠 𝑖𝑓h𝑒h𝑎𝑠 𝑗𝑢𝑠𝑡 h𝑏𝑜𝑢𝑔 𝑡 h𝑠 𝑜𝑒𝑠)𝑑𝑒𝑝𝑒𝑛𝑑𝑠𝑜𝑛

𝑝 ( h𝑀𝑖𝑐 𝑎𝑒𝑙𝑏𝑢𝑦𝑖𝑛𝑔𝑠𝑜𝑐𝑘𝑠 )𝑝 ( h𝑀𝑖𝑐 𝑎𝑒𝑙𝑏𝑢𝑦𝑖𝑛𝑔 h𝑠 𝑜𝑒𝑠 )

𝑝 ( 𝐴∨𝐵 )=𝑝 (𝐵∨𝐴 ) 𝑝 ( 𝐴 )𝑝 (𝐵 )

….Amazon (AMZN) calls this homegrown math "item-to-item collaborative filtering," and it's used this algorithm to heavily customize the browsing experience for returning customers…. Judging by Amazon's success, the recommendation system works. The company reported a 29% sales increase to $12.83 billion during its second fiscal quarter, up from $9.9 billion during the same time last year. A lot of that growth arguably has to do with the way Amazon has integrated recommendations into nearly every part of the purchasing process from product discovery to checkout.

http://tech.fortune.cnn.com/2012/07/30/amazon-5/

“In theory there is no difference between theory and practice; in practice, there is”.

Yogi Berra, cited in Nassim Taleb, Antifragile.

Big data techniques

NoSQL (ala MongoDB) Map-Reduce (e.g. Hadoop)

Embedded devices

Connected Devices

On Premise

Off Premise

Business Intelligence

Customers Employees, Partners

The Power of an Intelligent System

Modern Platform for the World’s Apps

Cloud OS

transforms the datacenterenables modern appsunlocks insights on any dataempowers people-centric IT

Cloud OS

flexible developmentunified dev-ops & managementcomplete data platformcommon identityintegrated virtualization

MICROSOFT

SERVICE PROVIDERON-PREMISES

1CONSISTENTPLATFORM

What Makes the Cloud OS Unique

RelationalNon-Relational Streaming

MANAGE ANY DATA, ANY SIZE, ANYWHERE

010101010101010101101010101010101001010101010101101010101010

Unified Monitoring, Management & Security

Data Movement

POLYBASE: COMBINING RELATIONAL AND NON-RELATIONAL DATAThe future of query processing

select... results set

Hadoop Data Warehouse

PolyBase

Single query for relational & Hadoop data

Process data in place

Future expansion to other data sources

Seamless: regular T-SQL command

19

20

Avoiding Lock-InWindows Virtual machines can move freely between all 3 clouds.

Windows Azure

Customer Data Center

Other Service ProvidersWindows

Virtual Machine

LocationOn-Premises On-Premises or

Service ProviderMicrosoft Cloud orService Provider

Rationale for Usage

Compliance

Scalability

Economies of Scale

Rapid Development

Complex, Legacy Applications

Compliance

Economics

TraditionalNON-VIRTUALIZED

AppliancePRIVATE

CloudPUBLIC

(Outside Firewall)

DATA PLATFORM DELIVERY MODELS

(Inside Firewall)

BALANCING ON PREMISE & CLOUDSnowline graph

A

Takeaways

1. “big data” can do some amazing stuff.2. Don’t think “big data” as much as “data needing non-

relational approaches”3. If your big data insights are probabilistic, which they often are,

have a plan to deal with variance. 4. Pick the most appropriate platform: Think “and” not “or”:

- Balance public cloud AND on-premise,- Combine “big data” with RDBMS.

Q+A