Upload
couchbase
View
266
Download
0
Embed Size (px)
Citation preview
Copyright © 2015 Criteo
Extreme Client Side Tweaking on Couchbase
Sébastien Foutrier
Lead Software Developer
June 3rd 2015
Copyright © 2015 Criteo 2
Criteo
• French start up since 2005• Keeping the spirit
• Machine Learning and Big Data oriented
• Algorithms as root
• Ad Tech business
• 230 in R&D 1500 worldwide• R&D teams in Paris
Copyright © 2015 Criteo 3
Criteo – Personalized performance advertising at scale
• Personalized• Per user per view • Real time product recommendation
• Performance• Cost of sales model• Cross device
• Scale• 994 millions unique users reach globally• 741 billions ads in 2014
Copyright © 2015 Criteo 4
Site Reliability Engineering at Criteo
50 DevOps engineers and growing with the company
Software infrastructure
Systems (CentOS, Windows, monitoring)
Automation (Chef)
Big Data infrastructure (Hadoop, Storm, Kafka, HBase)
NoSQL infrastructure (Memcached, Couchbase, Graphite,ElasticSearch, MongoDB, Cassandra, Redis)
Escalation
Predictive monitoring
Build & continuous integration
Copyright © 2015 Criteo 5
Hardware Infrastructure
• 6 Datacenters
• 10k+ server (50/50 windows/linux)
• Private worldwide dedicated network up to 10Gb/s
• One of the biggest Hadoop clusters in Europe(1k+ servers, 37 PB)
Copyright © 2015 Criteo 6
From SQL to some NoSQL
• Criteo started with• User data in client side cookies• Client and campaign configurations in MS SQL server
• First issues• RAM cache in IIS servers• Server to server calls
• Added some Memcached …. big instances
Copyright © 2015 Criteo 7
Next steps
• Real Time Bidding • 100 ms to respond
• Server to servers calls => bye bye client side cookies
• Additional challenge : multi datacenter => sync !
• Lots of instances
• Need for durability• Couchbase comes in
Copyright © 2015 Criteo 8
Grown big !
• 550 Couchbase servers (from 1.8 to 3)
• 24 clusters
• 107 TB Total RAM and SSD space
• 10M hits/sec worldwide
• 2M hits/sec peak single cluster
Copyright © 2015 Criteo 9
Things get serious, issues also
• 3.5K+ Clients
• Slow cross datacenter links (70ms)
• Blocking call
• 9K TCP sockets reached !• Client buffering blows memory/threads
Copyright © 2015 Criteo 10
Solutions ?
• Moxi• Unstable• Heavy CPU load• Bandwidth x3
• Prototype !• https://code.google.com/p/memcached/wiki/BinaryProtocolRevamped
Couchbase
Moxi Memcached
Couchbase
Moxi Memcached
Copyright © 2015 Criteo 11
Raw socket, from scratch
• Send(Byte[])• Synchronous call• request buffering• sending thread(s)
• BeginSend(…) / EndSend(…)• Overhead
• SendAsync(SocketAsyncEventArgs e)• Efficient when reusing ‘e’
Copyright © 2015 Criteo 12
The asynchronous API
if (!client.Get("Hello", (Status s, byte[] v) =>
{
if (s == Status.NoError && v != null)
Console.WriteLine("Hello, " + Encoding.UTF8.GetString(v));
else
Console.WriteLine("Get failed with status " + s);
}))
Console.WriteLine("Get failed ");
Copyright © 2015 Criteo 13
Asynchronous gain
• Less context switches• Less threads• Easy to pipeline !
Copyright © 2015 Criteo 14
TCP socket
Sequential model
Web Server Couchbase
Copyright © 2015 Criteo 15
TCP socket
Pipelining model
Web Server Couchbase
Copyright © 2015 Criteo 16
TCP socket
Socket pooling
TCP socket
Web Server Couchbase
Copyright © 2015 Criteo 17
TCP socket
TCP Windows
Web Server Couchbase
Copyright © 2015 Criteo 18
Couchbase
Redundancy (Memcached bucket)
Web Server
Couchbase
Couchbase
Couchbase
OKDrop
Z z zz
Copyright © 2015 Criteo 19
Couchbase
GetWithReplica (Couchbase bucket)
Web Server
Couchbase
Couchbase
Couchbase
OKOK
Z z zz
Copyright © 2015 Criteo 20
Benchmark
• Couchbase 1.Xusing Enyim.Caching
• Ping 277 ms(Paris-Tokyo)
• 20 nodes• 20 socket per node• 40 client threads
Copyright © 2015 Criteo 21
Benchmark
• Criteo.Memcache• Ping 277 ms
(Paris-Tokyo)• 20 nodes• 1 socket per node• 4 client threads
Copyright © 2015 Criteo 22
What is supported?
• Get/Set/Update/Add• GetAndTouch• Increment/Decrement• Ketama locator• Couchbase locator• GetWithReplica• Stats
• The features you’ll add!
Copyright © 2015 Criteo 23
Conclusion
• Open source (feel free to contribute):• https://github.com/criteo/memcache-driver
• Specific use cases• Async in Couchbase SDK 2.1
Copyright © 2015 Criteo 24
Conclusion (bis)
CRITEO IS HIRING
Copyright © 2015 Criteo
Thanks !Questions ?