Embed Size (px)
Citation preview
In a constantly changing industry you needa forward-thinking technology partner
Cassandra and Hybrid-Cloud: Mache
Neil Avery, CTOEmail: [email protected]: @avery_neil
2
Cassandra and the move the Hybrid cloud - [email protected]
Cloud is seen as a solution within financial services
Key Drivers: - Regulations, Cost, Data-Centre, Capacity- Running a Data-Centre is hard
Grid is the ‘Quick Win’
Projected AWS revenue in 2016: $1.8b
3
Cassandra and the move the Hybrid cloud - [email protected]
Compute Grids seem like a quick-win.
- Large scale, DC footprint
- Service most of the banks function
- Consume majority of DC estate
Scale: 10k-100k cores per app
4
Cassandra and the move the Hybrid cloud - [email protected]
Grid is a ‘quick win’ but…
- Cloud isn’t – public cloud sucks
- Security, compliance, outsourcing
- Hybrid Cloud helps…
6
Cassandra and the move the Hybrid cloud - [email protected]
Source: http://www.csc.com/cloud/insights/105069-hybrid_cloud
7
Cassandra and the move the Hybrid cloud - [email protected]
Hybrid Cloud Challenges affecting Grid
The key thing here is:
The network is different – leading to :
security, operations, integration challenges
The grid-application architecture will also be different
8
Cassandra and the move the Hybrid cloud - [email protected]
What do compute grids look like?
Typical numbers for a standard grid-app
- 30k cores/engine
- 50m tasks
- 7Mb per task
- 184GB peak data
- 10,500 GB daily
Numbers for a ‘data & compute’
- 18k cores/engines
- 7.5+GB per engine
- Peak: 144k GB – we can
share – 375 servers - or
385GB per server
9
Cassandra and the move the Hybrid cloud - [email protected]
Adopting grid + hybrid cloud = Data locality challenge
- 2 Cloud regions: 2 x (16 datacentres, 80k servers)
- 2 Datacentres (1-region)
- Cloud Exchange
- 3 locations for data to live
- Data has a life-cycle
- Thin-pipes == data-gravity - : 144k GB ?
10
Cassandra and the move the Hybrid cloud - [email protected]
The solution: Cassandra ‘DataCentres’
Mr Thin Pipes versus Cassandra
11
Cassandra and the move the Hybrid cloud - [email protected]
Mapping Cassandra onto Cloud regions
Cloud Region A
Source: http://docs.datastax.com/en/cassandra/2.2/cassandra/dml/dmlClientRequestsRead.html
Network Topology
Cloud Region B
Cloud Exchange
On-Prem
12
Cassandra and the move the Hybrid cloud - [email protected]
We need to solve the problem of
- Massive data
- Massive Compute (read-mostly)
- Thin pipes
- Data Pools
Be nice to the network
- NetworkTopologyStrategy
- LOCAL_QUORM
- REPLICATION_FACTOR
13
Cassandra and the move the Hybrid cloud - [email protected]
Cassandra allows us to shape the data layer
… but we have long-lived, high-read frequency, massive-data
‘16MB x 200 x 10k – read-mostly data’
Answer: Nearside cache
14
Cassandra and the move the Hybrid cloud - [email protected]
Why not use an IMDB? (Coherence, Gemfire, Gigaspaces)
- 16MB x 200 items per task
- 10k clients
- Leads to network DoS
IMDB’s only read from a primary-object- ‘Read’s don’t scale’
15
Cassandra and the move the Hybrid cloud - [email protected]
Introducing Mache….
Nearside-Cache = ‘mash-up of Open source’ htthttps://github.com/Excelian/Machetthttps://github.com/Excelian/MacheDistributed nearside-cache with eventing/invalidation
- Cassandra
- Google Guava Cache
- Spring Data
- Kafka, ActiveMq, Rabbit
- Yes – it’s java only
16
Cassandra and the move the Hybrid cloud - [email protected]
- Avoid heavy write loads:can cause event storms
- Read-mostly – data consistency- Infrequently changing data- Read with some write – Kafka
Mache: Use case
17
Cassandra and the move the Hybrid cloud - [email protected]
Mache: Performance environment
- Cassandra 2.x (4 nodes)
- Docker, Kubernetes, Fabric8,
- SDN: Project Calico
18
Cassandra and the move the Hybrid cloud - [email protected]
Cassandra: PerformanceCaveat: office network
19
Cassandra and the move the Hybrid cloud - [email protected]
Mache: PerformanceCaveat: office network
20
Cassandra and the move the Hybrid cloud - [email protected]
Mache: Update Frequency v LatencyCassandra, Mache & Kafka
21
Cassandra and the move the Hybrid cloud - [email protected]
Mache: Code example1) Cassandra Data
2) Creating the cache
3) Using the cache
22
Cassandra and the move the Hybrid cloud - [email protected]
Next steps
- Rest/JSON service
- JMX Support
- Nearside Overflow to disk
- Selective event to enable continuous query
- PoC’s
- Eventing API (listener)
Mache: available now!https://github.com/Excelian/Mache
23
Cassandra and the move the Hybrid cloud - [email protected]
Key takeaways:
• Cloud has found its home within FS
• Hybrid cloud represents new challenges
• Cassandra solves ‘data-gravity’ problem
• Nearside caching offloads high frequency reads from the storage
layer
https://github.com/Excelian/Mache
Cape Town Office1st Floor, Corporate Place
13 Mispel Road, BellvilleCape Town, 7530
South Africa+27(0) 21 944 9900
Frankfurt OfficeBrockenheimer Landstrasse 17/19
60325 FrankfurtGermany
+49 (0) 69 710455 183
London Office44 Featherstone Street
London, EC1Y 8RNUnited Kingdom
+44 (0)20 7336 9595
New York OfficeSuite 503, 100 Wall Street
New York, NY 10005
United States
Poland Officeul. Plac Konstytucji 3 Maja, 3, Silver
Tower
50-048 Wrocław
Poland
Sydney Office1 York Street
Level 2Sydney, NSW 2000
Australia+61 (2) 9191 7810
Toronto Office100 King Street West, Suite 5600
Toronto, Ontario M5X 1C9Canada
+1 (203) 295 5240
www.excelian.com
