Embed Size (px)
Citation preview
DISTRIBUTED COMPUTINGWITH HAZELCAST
www.hazelcast.com
DISCLAIMERThis presentation may include certain forward-looking statements and projections provided bythe Company. Any such statements and projections reflect various estimates and assumptionsby the Company concerning anticipated results. These statements involve risks, uncertainties
and assumptions and are based on the current estimates and assumptions of the managementof the Company as of the date of this presentation and are subject to uncertainty and changes.
No representations or warranties are made by the Company as to the accuracy of any suchstatements or projections. Whether or not any such forward-looking statements or projectionsare in fact achieved will depend upon future events some of which are not within the control of
the Company. Accordingly, actual results may vary from the projected results and suchvariations may be material. Statements contained herein describing documents and
agreements are summaries only and such summaries are qualified in their entirety by referenceto such documents and agreements.
www.hazelcast.com
THAT'S MEChris(toph) EngelbertTwitter: @noctarius2k8+ years of Java WeirdonessPerformance, GC, traffic topicsApache CommitterGaming, Travel Management, ...
www.hazelcast.com
OUR SPACE TRIP ROADMAPHazelcastDistributed ComputingDistributed ExecutorServiceEntryProcessorMap & ReduceQuestions
www.hazelcast.com
HAZELCASTPICKIN' DIAMONDS
www.hazelcast.com
WHAT IS HAZELCAST?In-Memory Data-GridData Partioning (Sharding)Java Collections ImplementationDistributed Computing Platform
www.hazelcast.com
WHY HAZELCAST?Automatic PartitioningFault ToleranceSync / Async BackupsFully DistributedIn-Memory for Highest Speed
www.hazelcast.com
WHY HAZELCAST?
www.hazelcast.com
www.hazelcast.com
WHY IN-MEMORYCOMPUTING?
www.hazelcast.com
TREND OF PRICES
Data Source: http://www.jcmit.com/memoryprice.htm
www.hazelcast.com
SPEED DIFFERENCE
Data Source: http://i.imgur.com/ykOjTVw.png
www.hazelcast.com
DISTRIBUTEDCOMPUTING
OR
MULTICORE CPU ON STEROIDS
www.hazelcast.com
THE IDEA OF DISTRIBUTED COMPUTING
Source: https://www.flickr.com/photos/stefan_ledwina/1853508040
www.hazelcast.com
THE BEGINNING
Source: http://en.wikipedia.org/wiki/File:KL_Advanced_Micro_Devices_AM9080.jpg
www.hazelcast.com
MULTICORE IS NOT NEW
Source: http://en.wikipedia.org/wiki/File:80386with387.JPG
www.hazelcast.com
CLUSTER IT
Source: http://rarecpus.com/images2/cpu_cluster.jpg
www.hazelcast.com
SUPER COMPUTER
Source: http://www.dkrz.de/about/aufgaben/dkrz-geschichte/rechnerhistorie-1
www.hazelcast.com
CLOUD COMPUTING
Source: https://farm6.staticflickr.com/5523/11407118963_e0e0870846_b_d.jpg
www.hazelcast.com
DISTRIBUTEDEXECUTORSERVICE
THE WHOLE CLUSTER IN YOUR HANDS
www.hazelcast.com
WHY A DISTRIBUTED EXECUTORSERVICE?j.l.Runnable / j.u.c.CallableOnly needs to be serializableSame Task all / multiple NodesShould not work on Data
www.hazelcast.com
Talk to me from all nodes
QUICK EXAMPLE
Runnable runnable = () -> println( "Hello from my Cluster-World" );IExecutorService es = hzInstance.getExecutorService( "default" );es.executeOnAllMembers( runnable );
www.hazelcast.com
DEMONSTRATION
www.hazelcast.com
ENTRYPROCESSORLOCKFREE DATA OPERATIONS
www.hazelcast.com
WHY ENTRYPROCESSOR?Prevents external LockingGuarantees AtomicityKind of a "Cluster-wide Thread-Safe"
www.hazelcast.com
Incrementing a counter atomically
QUICK EXAMPLE
private int increment( Map.Entry entry ) { val newValue = entry.getValue() + 1; entry.setValue( newValue ); return newValue;}
IMap map = hazelcastInstance.getMap( "default" );int newId = map.executeOnKey( "idgen", this::increment );
www.hazelcast.com
DEMONSTRATION
www.hazelcast.com
MAP & REDUCETHE BLACK MAGIC FROM PLANET GOOGLE
www.hazelcast.com
USE CASESLog AnalysisData QueryingAggregation and summingDistributed SortETL (Extract Transform Load)and more...
www.hazelcast.com
SIMPLE STEPSReadMap / TransformReduce
www.hazelcast.com
FULL STEPSReadMap / TransformCombiningGrouping / ShufflingReduceCollating
www.hazelcast.com
MAPREDUCE WORKFLOW
www.hazelcast.com
Data are mapped / transformed in a set of key-value pairs
SOME PSEUDO CODE (1/3)
MAPPING
map( key:String, document:String ):Void -> for each w:Word in document: emit( w, 1 )
www.hazelcast.com
Multiple values are combined to an intermediate result to preserve traffic
SOME PSEUDO CODE (2/3)
COMBINING
combine( word:Word, counts:List[Int] ):Void -> emit( word, sum( counts ) )
www.hazelcast.com
Values are reduced / aggregated to the requested result
SOME PSEUDO CODE (3/3)
REDUCING
reduce( word:String, counts:List[Int] ):Int -> return sum( counts )
www.hazelcast.com
FOR MATHEMATICIANSProcess: (K x V)* → (L x W)* ⇒ [(l1, w1), …, (lm, wm)]
Mapping: (K x V) → (L x W)* ⇒ (k, v) → [(l1, w1), …, (ln, wn)]
Reducing: L x W* → X* ⇒ (l, [w1, …, wn]) → [x1, …,xn]
www.hazelcast.com
MAPREDUCE PROGRAMS INGOOGLE SOURCE TREE
Source: http://research.google.com/archive/mapreduce-osdi04-slides/index-auto-0005.html
www.hazelcast.com
DEMONSTRATION
www.hazelcast.com
@noctarius2k@hazelcast
http://www.sourceprojects.comhttp://github.com/noctarius
THANK YOU!ANY QUESTIONS?
www.hazelcast.com
