Building a Business on Open Source Distributed Computing

8/14/2019 Building a Business on Open Source Distributed Computing

1/89

Building a Business on Open SourceDistributed Computing

company: www.visibletechnologies.com

blog: www.roadtofailure.comtwitter: @lusciouspear

Sunday, December 20, 2009
http://www.roadtofailure.com/http://www.roadtofailure.com/http://www.visibletechnologies.com/http://www.visibletechnologies.com/


2/89

Social Media and Scaling



3/89


Scalability Matters Now.



4/89



SM produces large, complex data



5/89




Anyone can collect the web



6/89





Make a Twitter in a few days



7/89






Easy to get TBs of data



8/89






Easy to get TBs of data

Big Data enabling new fields forcompanies



9/89

What Visible Does



10/89

What Visible Does

BI and Brand Management on SocialMedia



11/89

What Visible Does

BI and Brand Management on SocialMedia

Listen, Monitor, Engage



12/89Sunday, December 20, 2009


15/89

Old Product: RDBMS



16/89

Old Product: RDBMS

A few MSSQL servers on boxes



17/89

Old Product: RDBMS


Lots of ETL



18/89

Old Product: RDBMS


Lots of ETL

Several TB, inserts slow, deletes

impossible, random fail



19/89

Why RDBMS Bad



20/89

Why RDBMS Bad

Nonlinear scale cost



21/89

Why RDBMS Bad


Used as a storage abstraction



22/89

Why RDBMS Bad



Mainly Select, Join, Group, Count



23/89

Why RDBMS Bad



Mainly Select, Join, Group, CountSpecialized Scale-Out ones meh



24/89

Why RDBMS Bad




Impedance Mismatch - Try to be High-Throughput, Low-Latency



25/89

Why RDBMS Bad




Impedance Mismatch - Try to be High-Throughput, Low-Latency

Swiss-army knife, unstable,

transactions, advanced SQL, tuningSunday, December 20, 2009


26/89

Why OSS?



27/89

Why OSS?

Previously all MS



28/89

Why OSS?

Previously all MS

It exists!



29/89

Why OSS?

Previously all MS

It exists!Scaling + Licensing = No



30/89

Why OSS?

Previously all MS


Cant build a platform without source



31/89

Why OSS?

Previously all MS


Cant build a platform without source

Its Enterprise Now!



32/89

Goals for New Platform



33/89


Golden Timeline



34/89


Golden Timeline

Search/Analyze *any* data



35/89


Golden Timeline

Search/Analyze *any* dataLinear Cost



36/89


Golden Timeline


Not Hacked Together



37/89


Golden Timeline


Not Hacked Together

Collect the Social Internet



38/89

HOW TO SCALE



39/89

HOW TO SCALE

What makes you special?



40/89

HOW TO SCALE

What makes you special?What are you willing to sacrifice?



41/89

HOW TO SCALE

What makes you special?What are you willing to sacrifice?

How will you structure the data?



42/89

Avoiding Impedance Mismatch



43/89


Most problems can be divided intoHigh or Low latency



44/89



Get a lot of data eventually, or a littlenow



45/89



Get a lot of data eventually, or a littlenow

MapReduce vs. Sharding/Indexing



46/89

Ecosystem

Hadoop DFS

HBase

Hive

MapReduce

CascadingPig

Katta/App

lications

Zookeeper

Unstructured

Storage

Structured

Storage

Raw

Processing

Compiled

Processing



47/89

Simple Workflow

Collect SemanticAnalysis

UnstructuredAnalysis

Store in

HBase

StructuredAnalysis

Indexing

Pull

Indexes

Load/

Replicate

Shards Search

Store in

Hadoop

Hadoop

Hadoop +

HBase

Lucene+

Solr+

Katta



48/89

Unstructured Processing Cluster

CollectSemantic

Analysis

Unstructured

AnalysisInternet

XMLHTMLHBase

Records

Structured

Store



49/89

Hadoop + MR



50/89

Hadoop + MR

Special: Crunch web-scale data fast



51/89

Hadoop + MR


Sacrifice: Low-Latency, Transactions,Random Access, Updates



52/89

Hadoop + MR


Sacrifice: Low-Latency, Transactions,Random Access, Updates

Structure: Chunked flat files



53/89

Structured Processing Cluster

Store in

HBase

Structured

Analysis

IndexingStore in

HadoopHBase

Records

Unstructured

ClusterSearch

Cluster

Lucene Index ShardedLucene Index

Enriched Data



54/89

Document Structure

ContentID: 00BAC189

Title: Iron Maiden Rules

Body: I think Janick Gers is an amazing guitarist blah blah

PostDT: 20090718

ParentID: 0FDEADBEEF

Permalink: www.roadtofailure.com/post?=20

http://www.roadtofailure.com/post?=20http://www.roadtofailure.com/post?=20http://www.roadtofailure.com/post?=20


55/89

HBase



56/89

HBase

Special: Scalable random/sequential

access almost as fast as RDBMS



57/89

HBase

Special: Scalable random/sequentialaccess almost as fast as RDBMS

Sacrifice: Joins, Secondary Indexes,Transactions (kind of)



58/89

HBase

Special: Scalable random/sequentialaccess almost as fast as RDBMS

Sacrifice: Joins, Secondary Indexes,Transactions (kind of)

Structure: BigTable - column oriented


h l


59/89

Search Cluster

Pull

Indexes

Load/

ReplicateShards

Lucene

Indexes from

HDFS

Lucene

Indexes

Lucene

Indexes

Search


h


60/89

Search


l


61/89

Katta + Solr


S l


62/89

Katta + Solr

Special: Sharded search


K S l


63/89

Katta + Solr


Sacrifice: Consistency, high-throughput


K S l


64/89

Katta + Solr


Sacrifice: Consistency, high-throughput

Structure: Reverse index


BI


65/89

BI


BI


66/89

BI

Group, Sort, Filter, Count, Sum


BI


67/89

BI


Semi-additive (Avg) rare but not hard


BI


68/89

BI



MapReduce Jobs


BI


69/89

BI



MapReduce Jobs

Faceted Search


E l


70/89

Examples



71/89



72/89


Ch ll


73/89

Challenges


Ch ll


74/89

Challenges

Scaling Search


Ch ll


75/89

Challenges

Scaling Search

Understanding Latency


Ch ll


76/89

Challenges

Scaling Search

Understanding LatencyWhat do we need now? Can

customers wait for big data?


Ch ll


77/89

Challenges

Scaling Search

Understanding LatencyWhat do we need now? Can

customers wait for big data?

Monitoring


R R l f S li


78/89

Recap: Rules for Scaling


R R l f S li


79/89


RDBMS is not a Swiss-Army Knife


Recap: R les for Scaling


80/89



Know your sacrifices




81/89




Know your specialness




82/89





Know your data structure




83/89





Know your data structure

Ponder Latency


What Next?


84/89

What Next?


What Next?


85/89

What Next?

HBase Analytics?


What Next?


86/89

What Next?

HBase Analytics?

What would make a bank trust it


What Next?


87/89

What Next?

HBase Analytics?

What would make a bank trust it

Teach people to think about data



88/89

...


The End


89/89

The End

company: www.visibletechnologies.com

blog: www.roadtofailure.comtwitter: @lusciouspear

[email protected]
http://www.roadtofailure.com/http://www.roadtofailure.com/http://www.visibletechnologies.com/http://www.visibletechnologies.com/

Documents

Building a Business on Open Source Distributed Computing