C* Summit 2013: No Whistling Required: Cabs, Cassandra, and Hailo by Dave Gardner

#CASSANDRA13

Cassandra at Hailo

David Gardner | Architect @ Hailo

CASSANDRASUMMIT2013

#CASSANDRA13 CASSANDRASUMMIT2013

What is this talk about?

•  1,352 changed files with 235,413 additions and 47,487 deletions

•  7,429 commits

•  1,653 tickets completed

https://github.com/apache/cassandra/compare/cassandra-0.6.0...cassandra-1.2

https://github.com/apache/cassandra/blob/trunk/CHANGES.txt

0.6 to 1.2

Cassandra adoption at Hailo from three perspectives:

1.  Development

2.  Operational

3.  Management

What this talk is about

What is Hailo?

Hailo is The Taxi Magnet. Use Hailo to get a cab wherever you are, whenever you want.

•  The world’s highest-rated taxi app – over 10,000 five-star reviews

•  Over 500,000 registered passengers

•  A Hailo e-hail is accepted by a driver every four seconds around the world

•  Hailo operates in ten cities from Tokyo to Toronto in just over eighteen months of operation

What is Hailo?

•  Hailo is a marketplace that facilitates over $100M in run-rate transactions and is making the world a better place for passengers and drivers

•  Hailo has raised over $50M in financing from the world's best investors including Union Square Ventures, Accel, the founder of Skype (via Atomico), Wellington Partners (Spotify), Sir Richard Branson, and our CEO's mother, Janice

Hailo is growing

The history

The story behind Cassandra adoption at Hailo

Hailo launched in London in November 2011

•  Launched on AWS

•  Two PHP/MySQL web apps plus a Java backend

•  Mostly built by a team of 3 or 4 backend engineers

•  MySQL multi-master for single AZ resilience

Why Cassandra?

•  A desire for greater resilience – “become a utility” Cassandra is designed for high availability

•  Plans for international expansion around a single consumer app Cassandra is good at global replication

•  Expected growth Cassandra scales linearly for both reads and writes

•  Prior experience I had experience with Cassandra and could recommend it

The path to adoption

•  Largely unilateral decision by developers – a result of a startup culture

•  Replacement of key consumer app functionality, splitting up the PHP/MySQL web app into a mixture of global PHP/Java services backed by a Cassandra data store

•  Launched into production in September 2012 – originally just powering North American expansion, before gradually switching over Dublin and London

Development perspective

“Cassandra just works”

Dom W, Senior Engineer

Use cases

1.  Entity storage

2.  Time series data

CF = customers

126007613634425612: createdTimestamp: 1370465412 email: dave@cruft.co givenName: Dave familyName: Gardner locale: en_GB phone: +447911111111

Considerations for entity storage

•  Do not read the entire entity, update one property and then write back a mutation containing every column

•  Only mutate columns that have been set

•  This avoids read-before-write race conditions

CF = comms

2013-06-01: 55374fa0-ce2b-11e2-8b8b-0800200c9a66: {“to”:”dave@c… a48bd800-ce2b-11e2-8b8b-0800200c9a66: {“to”:”foo@ex… b0e15850-ce2b-11e2-8b8b-0800200c9a66: {“to”:”bar@ho … bfac6c80-ce2b-11e2-8b8b-0800200c9a66: {“to”:”baz@fo…

CF = comms

dave@cruft.co: 13b247f0-ce2c-11e2-8b8b-0800200c9a66: {“to”:”dave@c… 20f70a40-ce2c-11e2-8b8b-0800200c9a66: {“to”:”dave@c… 2b44d3b0-ce2c-11e2-8b8b-0800200c9a66: {“to”:”dave@c… 338a22f0-ce2c-11e2-8b8b-0800200c9a66: {“to”:”dave@c…

Considerations for time series storage

•  Choose row key carefully, since this partitions the records

•  Think about how many records you want in a single row

•  Denormalise on write into many indexes

Client libraries

•  Astyanax (Java)

•  phpcassa (PHP)

•  github.com/carloscm/gossie (Go)

Analytics

•  With Cassandra we lost the ability to carry out analytics eg: COUNT, SUM, AVG, GROUP BY

•  We use Acunu Analytics to give us this abilty in real time, for pre-planned query templates

•  It is backed by Cassandra and therefore highly available, resilient and globally distributed

•  Integration is straightforward

SELECT SUM(accepted), SUM(ignored), SUM(declined), SUM(withdrawn) FROM Allocations WHERE timestamp BETWEEN '1 week ago' AND 'now’ AND driver='LON123456789’ GROUP BY timestamp(day)

Challenges

10 Average years experience per team member

MySQL Cassandra

Lessons learned

Have an advocate

•  Get someone who will sell the vision internally

•  Make an effort to get everyone on board

Learn the theory

•  Teach each team member the fundamentals

•  CQL can encourage an SQL mindset, but it’s important to understand the underlying data model

•  Make a real effort to share knowledge – keep in mind the gulf in experience for most team members between their old world and the new world (SQL vs NoSQL)

•  Peer review data models

Operational perspective

“Allows a team of 2 to achieve things they wouldn’t have considered before Cassandra existed”

Chris H, Operations Engineer

2 clusters

6 machines per region

3 regions (stats cluster pending addition of third DC)

Operational Cluster

Stats Cluster

ap-southeast-1 us-east-1 eu-west-1

us-east-1 eu-west-1

AWS VPCs with Open VPN links

3 AZs per region

m1.large machines

Provisoned IOPS EBS

Operational Cluster

Stats Cluster

~ 600GB/node

~ 100GB/node

Backups

•  SSTable snapshot

•  Used to upload to S3, but this was taking >6 hours and consuming all our network bandwidth

•  Now take EBS snapshot of the SSTable snapshots

Encryption

•  Requirement for NYC launch

•  We use dmcrypt to encrypt the entire EBS volume

•  Chose dmcrypt because it is uncomplicated

•  Our tests show a 1% performance hit in disk performance, which concurs with what Amazon suggest

Datastax Ops Centre

•  We run the free version

•  Offers up easily accessible “one screen” overviews of the activity of the entire cluster

•  Big fans – an easy win

Multi DC

•  Something that Cassandra makes trivial

•  Would have been very difficult to accomplish active-active inter-DC replication with a team of 2 without Cassandra

•  Rolling repair needed to make it safe (we use LOCAL_QUORUM)

•  We schedule “narrow repairs” on different nodes in our cluster each night

Compression

•  Our stats cluster was running at ~1.5TB per node

•  We didn’t want to add more nodes

•  With compression, we are now back to ~600GB

•  Easy to accomplish

•  `nodetool upgradesstables` on a rolling schedule

Lessons learned

Management perspective

“The days of the quick and dirty are over”

Simon V, EVP Operations

Technically, everything is fine…

•  Our COO feels that C* is “technically good and beautiful”, a “perfectly good option”

•  Our EVPO says that C* reminds him of a time series database in use at Goldman Sachs that had “very good performance”

…but there are concerns

People who can attempt to query MySQL

People who can attempt to

query Cassandra

Lessons learned

Keep the business informed

•  Pre-launch, we were tasked with increasing resiliency

•  Cassandra addressed immediate business needs, but the trade offs involved should have been communicated more clearly

Sing from the same hymn sheet

•  A senior founding engineer had doubts about the adoption of Cassandra until very recently

•  In the presence of business doubt, this lack of consistency amongst developers exacerbated the concerns

•  We should have made more effort to make bilateral decisions on adoption – I don’t think this would have been hard to achieve

Provide solutions

•  There are many options for ad-hoc querying of Cassandra

•  We underestimated the impact of not having a good solution for this from the very beginning

People who can attempt to query MySQL People who can

attempt to query Cassandra

Conclusions

We like Cassandra

•  Solid design

•  HA characteristics

•  Easy multi-DC setup

•  Simplicity of operation

Lessons for successful adoption

•  Have an advocate, sell the dream

•  Learn the fundamentals, get the best out of Cassandra

•  Invest in tools to make life easier

•  Keep management in the loop, explain the trade offs

The future

•  We will continue to invest in Cassandra as we expand globally

•  We will hire people with experience running Cassandra

•  We will focus on expanding our reporting facilities

•  We aspire to extend our network (1M consumer installs, wallet) beyond cabs

•  We will continue to hire the best engineers in London, NYC and Asia

#CASSANDRA13

Thank you

CASSANDRASUMMIT2013

C* Summit 2013: No Whistling Required: Cabs, Cassandra, and Hailo by Dave Gardner

Technology

Whistling Kettle Menu

Say Hailo to the future of taxis

Japan Venture Environment and Hailo

Cabs Standard Cabs Service Cabs ... - Capital Elevator Ltd · Elevator Cabs Everything you need or we will do it ! Standard Cabs Service Cabs Custom Cabs Panoramic Cabs Elevator Cabs

How Hailo fuels its growth using NoSQL storage and analytics - Dave Gardner (Hailo)

The Whistling Thorn - Isobel Chace ###

Black-bellied Whistling Ducks

Plumed Whistling Duck.pdf

Driver Handbook - Hailo Jobs

on the move - a2b Australia€¦ · Sydney Taxis Combined Services, Silver Service, Apple Taxis, 13LIME, ABC Cabs, South Western Cabs 3,931 Melbourne 13 CABS, Black Cabs, North Suburban

Cabs In Jaipur

IWP Hailo Catalogue

Cabs, Cassandra, and Hailo (at Cassandra EU)

Metro city cabs

Gebrauchs- und Bedienungsanleitung DE · Manual de utilizare Scară telescopică din aluminiu Hailo FlexLine Lietošanas un ekspluatācijas instrukcija Hailo FlexLine alumīnija teleskopiskās

UK Parliament · LICENSED LONDON TAXI RECEIPT Transport for London AMOUNT £ ALWAYS USE TRADITIONAL LONDON TAXI-CABS THANK YOU FOR YOUR BUSINESS Next time, choose Hailo Receipt Be

Acunu Analytics and Cassandra at Hailo All Your Base 2013

HaILO WORK and WELFaRE REFORM - STICERDsticerd.lse.ac.uk/dps/case/cr/casereport81.pdf · 8 • Hailo Work and Welfare reform 9• It’s been really difficult because we don’t have

TurbomachineryConditionMonitor CABS

Richmond Grace - The Whistling Mother